Skip to content

Store only immediate descendants; compute transitive on demand#753

Draft
st0012 wants to merge 1 commit intomainfrom
immediate-only-descendants
Draft

Store only immediate descendants; compute transitive on demand#753
st0012 wants to merge 1 commit intomainfrom
immediate-only-descendants

Conversation

@st0012
Copy link
Copy Markdown
Member

@st0012 st0012 commented Apr 16, 2026

Summary

Replaces per-namespace transitive-descendant storage with immediate-only edges. Transitive descendants are now computed on demand via a BFS iterator over the direct-child DAG.

Previously, each Namespace stored its full transitive descendant set, populated during ancestor linearization by walking every class's ancestor chain and inserting into each ancestor's set. Heavy mixin ancestors (Object, Kernel, Module, ActiveSupport::Concern, etc.) accumulated tens of thousands of descendant entries each.

After this change, each namespace stores only its direct children (via superclass, include, prepend, or — for singleton classes — extend). A new Graph::transitive_descendants(root) iterator does a BFS with visited-set dedup to recover the transitive set on demand.

External API behavior is preserved: the MCP get_descendants tool and the FFI rdx_declaration_descendants iterator return the same sets as before (root included first, matching prior stored-set semantics).

Performance

Three runs per side on an internal Ruby monorepo (1.47M declarations, 1.63M definitions). Release build, same machine, back-to-back runs, stable power.

Metric Main (3 runs) Branch (3 runs) Δ (median)
Resolution phase 42.75s / 41.91s / 40.39s 29.07s / 27.27s / 27.65s −14.26s (−34.0%)
Total indexing 52.45s / 51.86s / 49.42s 39.13s / 35.90s / 36.45s −15.41s (−29.7%)
Max RSS 4688 / 4931 / 5392 MB 4818 / 5443 / 5338 MB inconclusive

Timing: robust improvement. Every branch run beat every main run by ≥10s end-to-end, with ~3s spread on each side and no overlap. The resolution-phase win is the dominant effect: eliminating propagate_descendants — which wrote to every ancestor on every recursion — takes the edge-write cost from O(ancestor_chain_depth) per class to O(1).

Memory: inconclusive from benchmarks. RSS varies by ~700 MB between back-to-back runs on either side (OS / file-system cache effects), which swamps any delta a 3-sample comparison could resolve. The theoretical storage reduction is clear — previously O(N × avg_ancestor_chain_depth) descendant entries, now O(N × avg_direct_parents) — but quantifying it would need heap profiling or many more samples under controlled conditions.

Declaration and definition counts are identical across sides (no semantic drift).

Design notes

  • Edge recording lives at the call sites in linearize_parent_class, linearize_mixins, and the singleton branch of linearize_parent_ancestors. Each call site invokes a small Resolver::record_immediate_descendant(parent_id, child_id) helper right before walking into the parent/mixin. The helper is idempotent and skips self-edges (handles cyclic self-inclusion like module A; include A; end).
  • linearize_ancestors is unchanged in shape — it does not take a child parameter. Recording at the call sites keeps the edge responsibility with the relationship being established, and naturally avoids recording edges during read-only traversals like search_ancestors.
  • Unresolved mixins still push Ancestor::Partial without calling linearize_ancestors, so no edge is recorded against unknown parents — matches prior behavior.
  • Invalidation Site C (remove-self-from-ancestors) collects 8-byte DeclarationIds instead of cloning the full Vec<Ancestor>.
  • Four new unit tests cover iterator invariants (root first, dedup across diamond paths, leaf termination, include chain) plus a dedicated direct-only storage test with a negative assertion that guards against regression to transitive storage.

@st0012 st0012 requested a review from a team as a code owner April 16, 2026 20:34
@st0012 st0012 marked this pull request as draft April 16, 2026 20:35
@st0012 st0012 force-pushed the immediate-only-descendants branch from e7d7900 to b451fa7 Compare April 16, 2026 21:01
Each Namespace previously stored its full transitive descendant set,
populated during ancestor linearization by walking every class's ancestor
chain and inserting into each ancestor's descendants set. Heavy mixin
ancestors like Object, Kernel, Module, and ActiveSupport::Concern
accumulated tens of thousands of entries each.

This change stores only immediate children (direct superclass, include,
prepend, or — for singleton classes — extend). Transitive descendants are
computed on demand via a BFS iterator over the direct-child DAG with
visited-set dedup. External API behavior is preserved: the MCP
get_descendants tool and the FFI rdx_declaration_descendants iterator
return the same sets as before.

Impact (measured on an internal Ruby monorepo, 1.47M declarations /
1.63M definitions):

  Max RSS:     5568.58 MB → 4884.39 MB  (-684 MB, -12.3%)
  Total index: 53.05s     → 42.26s      (-20.3%)
  Resolution:  43.01s     → 30.71s      (-28.6%)

Resolution time drops because propagate_descendants — which wrote to
every ancestor on every recursion — is gone. The write cost is now
O(1) per class instead of O(ancestor_chain_depth).
@st0012 st0012 force-pushed the immediate-only-descendants branch from b451fa7 to ae7a582 Compare April 17, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant