IGNORE: base for enablement

CLAUDE.md

@@ -25,6 +25,7 @@
 - Leave resources open (terminate threads, close files)
 - Make breaking public API changes
 - Use `sleep` in tests for synchronization (use deterministic waits: Queue, ConditionVariable, flush methods that block, or mock time)
+- Use non-verifying `double()` in tests — always use `instance_double(ClassName)`, `class_double(ClassName)`, or `object_double(instance)` so RSpec verifies method existence and arity against the real class
 
 ## Ask First
 
@@ -66,6 +67,17 @@ yamllint --strict .github/workflows/your-workflow.yml
 actionlint .github/workflows/your-workflow.yml
 ```
 
+## Troubleshooting
+
+When investigating a bug or unexpected behavior:
+1. State the observed behavior clearly
+2. Formulate a specific, testable hypothesis
+3. Verify the hypothesis with evidence (run code, read logs, add instrumentation) before proceeding
+4. Do not guess at causes or propose fixes until the hypothesis is confirmed
+5. If the hypothesis is disproven, formulate a new one — do not stack speculations
+6. When the fix is found, document in learnings: observed behavior, hypothesis chain, root cause, and fix
+7. Every hypothesis verification must become a test — if you checked behavior manually, encode that check as a spec so the expected behavior is retained in the test suite
+
 ## Code Changes
 
 - Read files before editing them

Steepfile

@@ -87,6 +87,8 @@ target :datadog do
   ignore 'lib/datadog/di/transport/http/api.rb'
   ignore 'lib/datadog/di/transport/http/diagnostics.rb'
   ignore 'lib/datadog/di/transport/http/input.rb'
+  ignore 'lib/datadog/symbol_database/component.rb'
+  ignore 'lib/datadog/symbol_database/extractor.rb'
   # steep thinks the type of the class is 'self', whatever that is,
   # and then complains that this type doesn't have any methods including
   # language basics like 'send' and 'raise'.

docs/DynamicInstrumentation.md

@@ -292,8 +292,9 @@ The value will fall back to default serialization.
 
 ## What Data Is Captured
 
-When a probe fires, Dynamic Instrumentation captures a snapshot of
-application state and sends it to Datadog. The snapshot includes:
+Dynamic instrumentation sends some of the application data to Datadog.
+
+**Probe snapshots** (captured when probes fire):
 
 - **Variable values** — local variables, method arguments, and return
   values, subject to the capture depth and collection size limits
@@ -311,6 +312,141 @@ application state and sends it to Datadog. The snapshot includes:
     redacted.
 - **Stack traces** — the call stack at the point the probe fires.
 
+**Symbol Database** (uploaded once at startup, see below):
+
+- Class, module, and method names from user application code
+- Method parameter names (not values)
+- Source file paths and line ranges
+- File content hashes (for source code version matching)
+- No runtime values, variable contents, or application data
+
+## Symbol Database
+
+The Symbol Database powers auto-completion in the Dynamic Instrumentation
+UI. When enabled, the tracer extracts symbol information (classes,
+modules, methods, parameters) from your running application and uploads
+it to Datadog via the Agent. This allows the DI UI to suggest class
+names, method names, and method parameters when creating probes.
+
+### Enabling the Symbol Database
+
+Symbol Database upload is enabled by default when Dynamic Instrumentation
+is enabled. No additional configuration is required. It activates via
+Remote Configuration when the DI UI is opened for your service.
+
+To explicitly disable it:
+
+    export DD_SYMBOL_DATABASE_UPLOAD_ENABLED=false
+
+For testing without Remote Configuration:
+
+    export DD_INTERNAL_FORCE_SYMBOL_DATABASE_UPLOAD=true
+
+### What Is Extracted
+
+The Symbol Database extracts metadata about your application's structure.
+It does **not** extract runtime values, variable contents, or any data
+that flows through your application.
+
+**Extracted:**
+- Class and module names
+- Method names and parameter names (not values)
+- Method visibility (public, private, protected)
+- Class inheritance (`superclass`) and module inclusion
+  (`included_modules`, `prepended_modules`)
+- Class variables and constants (names only, not values)
+- Source file paths and line ranges
+- File content hashes (Git-compatible SHA-1, for commit inference)
+
+**Note on generated methods:** Methods generated by `attr_writer`,
+`attr_accessor`, ActiveRecord associations, and similar metaprogramming
+are **not extracted**. Their `source_location` points to gem or stdlib
+code (e.g. `activerecord/lib/...`, `forwardable.rb`), so they are
+filtered out along with other non-user code. Only methods whose source
+is in your application files appear in autocomplete.
+
+**Not extracted:**
+- Instance variable names or values
+- Local variable names or values
+- Method return types (Ruby is dynamically typed)
+- Runtime data of any kind
+
+### What Is Uploaded
+
+Symbol data is uploaded to the Datadog Agent as compressed JSON via the
+`/symdb/v1/input` endpoint. The Agent forwards it to the Datadog
+backend. Uploads occur once at startup (after Remote Configuration
+enables the feature) and are deduplicated — the same symbols are not
+re-uploaded unless the application restarts.
+
+### Which Code Is Included
+
+Only **user application code** is extracted. The following are
+automatically excluded:
+
+- All installed gems (detected via `/gems/` in the source path)
+- Ruby standard library
+- The Datadog tracer itself (`Datadog::` namespace)
+- Test code (`/spec/`, `/test/` paths)
+- Code loaded via `eval()`
+
+This means internal or private gems installed via Bundler are also
+excluded. There is currently no mechanism to force-include specific
+gems.
+
+### Behavior Differences from Other Tracers
+
+Ruby's Symbol Database implementation differs from Java, Python, and
+.NET in several ways:
+
+#### Scope hierarchy
+
+Ruby uses `FILE` as the root scope type (one per source file), with
+`CLASS` or `MODULE` scopes nested inside. Java uses `JAR`, .NET uses
+`ASSEMBLY`, and Python uses `MODULE` (one per Python module file).
+Within each root scope, all tracers extract `CLASS` and `METHOD` scopes.
+
+#### Code filtering
+
+Java, Python, and .NET ship curated lists of known third-party package
+names (600+ to 5,000 entries) and support `DD_THIRD_PARTY_DETECTION_EXCLUDES`
+to force-include specific libraries. Ruby uses path-based filtering
+(`/gems/`, `/ruby/`) instead, which is effective for Ruby's gem
+ecosystem but does not support overrides. The
+`DD_THIRD_PARTY_DETECTION_INCLUDES` and `DD_THIRD_PARTY_DETECTION_EXCLUDES`
+environment variables are not yet implemented for Ruby.
+
+#### Deferred features
+
+The following features available in other tracers are not yet
+implemented for Ruby:
+
+- **Instance variable extraction** (FIELD symbols) — Java and .NET
+  extract class fields; Ruby would require runtime introspection or
+  source parsing
+- **Local variable extraction** (LOCAL scopes) — Java and .NET extract
+  local variables from bytecode/PDB debug info; not available via Ruby
+  introspection
+- **Closure/block scopes** — .NET extracts lambda and async closure
+  scopes; Ruby blocks, procs, and lambdas are not yet extracted
+- **Payload splitting** — Java splits uploads exceeding 50 MB into
+  smaller chunks; Ruby skips the upload entirely if it exceeds 50 MB
+  (unlikely for typical applications)
+- **Fork deduplication** — Python coordinates uploads across forked
+  workers (Gunicorn, uWSGI); Ruby does not yet deduplicate uploads in
+  preforking servers (Puma clustered mode, Unicorn, Passenger), meaning
+  each worker uploads independently
+- **Injectable line information** — Go and .NET report which lines
+  within a method can accept probes; Ruby does not include this metadata
+
+#### Class methods
+
+Class methods (`def self.foo`) are extracted but **not uploaded** by
+default. Ruby's Dynamic Instrumentation can only instrument instance
+methods (via `prepend`), so including class methods would present
+completions for methods that cannot be probed. This may change when DI
+gains singleton class instrumentation support.
+
 ## Rate Limiting and Performance
 
 ### Default Rate Limits

docs/class_methods_di_design.md

@@ -0,0 +1,190 @@
+# Class Methods in Symbol Database
+
+## What Are Class Methods in Ruby?
+
+In Ruby, class methods are defined on the singleton class of the class object:
+
+```ruby
+class User
+  def digest(string)       # instance method — User#digest
+    BCrypt::Password.create(string)
+  end
+
+  def self.digest(string)  # class method — User.digest
+    BCrypt::Password.create(string)
+  end
+end
+```
+
+Both can coexist with the same name. They are completely separate methods accessed
+through different lookup chains.
+
+## Cross-Language Equivalents
+
+| Language | Equivalent | Same name as instance method possible? | DI support |
+|----------|-----------|---------------------------------------|-----------|
+| Ruby | `def self.foo` (singleton method) | Yes | No (see below) |
+| Java | `static` method | Yes — resolves via `INVOKEVIRTUAL` vs `INVOKESTATIC` | Yes |
+| C# (.NET) | `static` method | Yes | Yes |
+| JavaScript | `static foo()` | Yes | Yes |
+| Python | `@classmethod` / `@staticmethod` | No — second definition overwrites first | Yes |
+| Go | Package-level function | Not applicable (no classes) | Yes (functions) |
+
+Java, C#, and JavaScript all support same-name instance + class methods and DI
+instruments both. Python avoids the naming collision entirely.
+
+## Why Ruby Class Methods Are Not Uploaded by Default
+
+Ruby DI instruments methods by prepending a module to a class's instance method
+lookup chain:
+
+```ruby
+cls.prepend(instrumentation_module)
+```
+
+This intercepts calls to **instance methods** (`cls.instance_method(:foo)`).
+It does **not** affect the singleton class. To instrument a class method, DI
+would need:
+
+```ruby
+cls.singleton_class.prepend(instrumentation_module)
+```
+
+This is not currently implemented in `lib/datadog/di/instrumenter.rb` — it only
+calls `cls.instance_method(method_name).source_location` (line 104) and never
+touches the singleton class.
+
+**Consequence:** Including class method scopes in symdb payloads would present
+completions in the DI UI for methods that cannot be probed. This is misleading
+and potentially confusing for users.
+
+## Backend Disambiguation: Java Static vs Instance Methods
+
+For languages where the same method name can exist as both instance and static,
+the probe specification uses the `signature` field in `Where` (Java's probe location):
+
+```java
+// com.datadog.debugger.probe.Where
+String typeName;    // "com.example.User"
+String methodName;  // "digest"
+String signature;   // "(Ljava/lang/String;)Ljava/lang/String;" — JVM descriptor
+```
+
+The JVM method descriptor encodes parameter and return types. Since static and
+instance methods both appear in the class's method table but with different
+descriptors (static methods don't have an implicit `this`), the signature
+disambiguates them.
+
+For Ruby, `method_type: "class"` in `language_specifics` serves this purpose once
+DI supports class method instrumentation.
+
+## Current Implementation
+
+Class methods are extracted but **gated behind an internal setting**:
+
+```ruby
+# Default: false — class methods not uploaded
+Datadog.configuration.symbol_database.internal.upload_class_methods
+
+# Or via env var (internal use only):
+DD_INTERNAL_SYMBOL_DATABASE_UPLOAD_CLASS_METHODS=true
+```
+
+When enabled, class methods are emitted as `METHOD` scopes with:
+- `name: "method_name"` (bare name, no `self.` prefix)
+- `language_specifics.method_type: "class"`
+
+The bare name (no `self.` prefix) matches Java/C# conventions. The
+`method_type: "class"` field disambiguates from instance methods with the
+same name — this is the standard cross-language approach used by all other
+Datadog tracers.
+
+## Path to Enabling Class Methods
+
+1. Implement singleton class instrumentation in `lib/datadog/di/instrumenter.rb`:
+   - Detect `method_type: "class"` in probe definition
+   - Use `cls.singleton_class.prepend(...)` instead of `cls.prepend(...)`
+   - Use `cls.singleton_class.instance_method(name)` for source location lookup
+
+2. Switch default to `true` and move setting from `internal` to public:
+   ```ruby
+   option :upload_class_methods do |o|
+     o.type :bool
+     o.default true  # once DI instruments class methods
+   end
+   ```
+
+3. The backend already stores `method_type` and can use it for DI UI completions
+   once the tracer can deliver on the probe.
+
+## Probe Spec Disambiguation (UI → Tracer via RC)
+
+The probe specification sent from the backend to the tracer via Remote Config uses
+`MethodProbeLocation` (TypeScript type in web-ui):
+
+```typescript
+// packages/api/endpoints/live-debugger/types/probe/probe-location.types.ts
+type MethodProbeLocation = {
+    typeName: string;    // e.g. "User"
+    methodName: string;  // e.g. "digest"
+    signature?: string;  // e.g. "String(Number, Object)" — optional
+};
+```
+
+There is **no `isClassMethod` or `isStatic` boolean** in the probe spec. For Java,
+disambiguation relies on the `signature` field: since static and instance methods have
+different JVM descriptors (static omits the implicit `this` parameter), the tracer can
+match the signature to the bytecode `MethodNode.access & Opcodes.ACC_STATIC`.
+
+**For Ruby, this approach doesn't work** because Ruby methods are untyped — a class
+method `def self.digest(string)` and instance method `def digest(string)` have
+identical `Method#parameters` output: `[[:req, :string]]`. There is no signature
+to distinguish them.
+
+**When Ruby DI adds class method support**, either:
+1. A new boolean field must be added to `MethodProbeLocation` (e.g. `isClassMethod`)
+2. Or the `signature` field is repurposed with a Ruby-specific convention
+
+This requires coordination between the web-ui, backend probe spec, and Ruby tracer.
+
+## `self` as an Implicit Argument
+
+Ruby DI emits `self` as the first `ARG` symbol for **instance methods** only.
+`self` is not in `Method#parameters` (it's implicit), but it must be registered so
+DI expression language can evaluate `self.name`, `self.class`, etc. at a probe point.
+
+For **class methods**, `self` is the class object itself — still accessible but less
+useful for DI expression evaluation, and not emitted to keep parity with other tracers.
+
+```ruby
+# In extract_method_parameters (extractor.rb):
+self_arg = if method_type == :instance
+  [Symbol.new(symbol_type: 'ARG', name: 'self', line: UNKNOWN_MIN_LINE)]
+else
+  []  # class methods: self not emitted
+end
+```
+
+## UI: How the Frontend Surfaces Methods
+
+The frontend uses these symdb API endpoints (web-ui/packages/api/endpoints/live-debugger/):
+- `/api/unstable/symdb-api/scopes/search` — search by class/method name
+- `/api/unstable/symdb-api/completions/scope/method` — get completions for a method probe
+
+The `DebuggerSymbolApi` type returned from search does NOT include `method_type` — the
+`LanguageSpecifics` type exposed to the frontend has `accessModifiers`, `annotations`,
+`interfaces`, `superClasses`, `returnType`, but no `method_type` or `isStatic`.
+
+**Implication:** Even if we upload class methods, the UI currently cannot distinguish
+them from instance methods in the search results. The `method_type: "class"` field
+is stored in the backend database but not surfaced to the frontend. Surfacing it would
+require a frontend change to `LanguageSpecifics` and UI rendering logic.
+
+## References
+
+- `lib/datadog/di/instrumenter.rb:104` — current instance-method-only lookup
+- `lib/datadog/symbol_database/extractor.rb` — `extract_singleton_method_scope`, `extract_method_parameters`
+- `lib/datadog/symbol_database/configuration/settings.rb` — `upload_class_methods` setting
+- `debugger-backend/debugger-common/.../TracerVersionChecker.kt` — language min versions
+- `web-ui/packages/api/endpoints/live-debugger/types/probe/probe-location.types.ts` — probe spec
+- `web-ui/packages/api/endpoints/live-debugger/types/symdb-scopes.types.ts` — LanguageSpecifics type