Analysis module performance: reduce allocs, CPU, and DB round-trips

[rh-jfuller]

graph analysis module had several performance inefficiencies:

1. Per-node JSON serialization in filter() — every node scanned during a GraphQuery::Query built serde_json::Value maps for all purls and CPEs even if query only needed substring matching
2. Deep cloning in BaseSummary — every node in the result tree cloned all purls, CPEs, and string fields from the graph into owned types
3. Double hash lookup in Context::intern — contains_key + get instead of single entry() call during graph construction
4. Per-node HashSet<Relationship> clone — the relationship filter set was cloned for every matching node in the async closure
5. Two seq DB queries in resolve_rh_external_sbom_descendants — first fetched the checksum value, then searched for matches
6. No caching for resolve_external_sbom — every ExternalNode hit during descendant traversal triggered a fresh DB query, even for duplicate references

Addressing these positively impacts:

• Memory: eliminates tens of MB transient heap allocs per query on large graphs (JSON construction), plus ~500 bytes per result node (Arc sharing vs deep clone)
• CPU: common filter path avoids all JSON serialization
• I/O: saves a DB round-trip per RH external node; eliminates all duplicate external resolution queries per request

No API changes. No schema changes. No migration required.

Summary by Sourcery

Improve performance of the analysis module by reducing per-node allocations and database round-trips, primarily through caching, shared ownership of summary data, and more efficient query and filtering paths.

New Features:

• Introduce a request-scoped cache for external SBOM resolution to deduplicate database lookups across descendant traversal.

Enhancements:

• Optimize RH external SBOM descendant resolution by replacing two sequential checksum queries with a single self-join query.
• Avoid unnecessary JSON construction in graph filtering by only materializing nested purl and CPE structures when the query string requires them.
• Reduce per-node memory usage in BaseSummary by sharing purl, CPE, and string fields via Arc instead of cloning.
• Improve string interning performance during graph loading by using a single hashmap entry operation instead of separate contains/get steps.
• Avoid cloning the relationship filter set for each node by sharing it across tasks via Arc.

Tests:

• Update analysis service tests to reflect shared purl and CPE storage using Arc-backed slices.

Summary by Sourcery

Improve analysis performance by reducing transient allocations, sharing immutable data, and eliminating redundant database queries.

Enhancements:

• Reduce analysis query allocations and CPU usage by avoiding unnecessary nested purl and CPE materialization during filtering.
• Share graph-derived summary data through reference-counted ownership to reduce per-node memory usage.
• Improve graph loading and concurrent traversal efficiency through streamlined string interning and shared relationship filters.
• Deduplicate external SBOM lookups within each request using a concurrent resolution cache.
• Resolve RH external SBOM descendants with a single checksum-matching database query instead of sequential round-trips.

Tests:

• Update analysis service assertions for Arc-backed purl and CPE collections.

[sourcery-ai]

Reviewer's Guide

This PR optimizes the analysis module by reducing per-node allocations, CPU overhead in filtering and summarization, and duplicate database lookups, primarily via smarter data sharing, conditional JSON construction, and request-scoped caching for external SBOM resolution.

Sequence diagram for cached external SBOM resolution

sequenceDiagram
    participant Collector
    participant ExternalSbomCache
    participant OnceCell
    participant Database as resolve_external_sbom

    Collector->>ExternalSbomCache: resolve(node_id, connection)
    ExternalSbomCache->>ExternalSbomCache: cache.lock()
    ExternalSbomCache->>ExternalSbomCache: cache.entry(node_id).or_default()
    ExternalSbomCache->>OnceCell: get_or_try_init(async { resolve_external_sbom(node_id, connection) })
    alt first caller for node_id
        OnceCell->>Database: resolve_external_sbom(node_id, connection)
        Database-->>OnceCell: Option<ResolvedSbom>
        OnceCell-->>ExternalSbomCache: Arc<Option<ResolvedSbom>>
    else cached result
        OnceCell-->>ExternalSbomCache: Arc<Option<ResolvedSbom>>
    end
    ExternalSbomCache-->>Collector: Option<ResolvedSbom>
    note over Collector,ExternalSbomCache: Concurrent calls for the same node_id share a single DB query via OnceCell

Loading

File-Level Changes

Change	Details	Files
Replace two-step checksum lookup for RH external SBOM descendants with a single self-join query and adjust downstream mapping.	Remove initial query that fetched checksum value for a specific node and SBOM Introduce ChecksumMatch FromQueryResult struct and raw SQL self-join over sbom_node_checksum to fetch matching nodes in other SBOMs Update filtering logic to use matched_node_id/matched_sbom_id fields instead of original node_id/sbom_id Adjust ResolvedSbom construction to use matched_* fields	modules/analysis/src/service/mod.rs
Avoid unconditional per-node JSON construction in GraphQuery::Query filter by only building nested purl/cpe JSON when the query string actually references them.	Inspect query.q for "purl:" and "cpe:" markers to decide whether nested JSON is needed Always expose borrowed purl and cpe collections via put_hidden for substring matching on flat fields Conditionally build serde_json::Value representations for purls and cpes only when required and add them to the context	modules/analysis/src/service/mod.rs
Introduce request-scoped ExternalSbomCache to deduplicate resolve_external_sbom calls across descendants and share it across Collector instances.	Add ExternalSbomResult/ExternalSbomCell/ExternalSbomMap aliases and ExternalSbomCache struct backed by Arc<Mutex<...>> Implement ExternalSbomCache::resolve using OnceCell::get_or_try_init to coalesce concurrent lookups per node_id Thread ExternalSbomCache through Collector and AnalysisService graph collection paths Use external_sbom_cache.resolve in descendant traversal instead of direct resolve_external_sbom calls	modules/analysis/src/service/collector.rs modules/analysis/src/service/mod.rs
Reduce cloning in BaseSummary by using Arc-based shared collections/strings and a small helper for defaulting optional Arc fields.	Change BaseSummary.purl/cpe to Arc<[Purl]> and Arc<[Cpe]> and document_id/product_name/product_version to Arc with schema annotations Define EMPTY_ARC_STRING LazyLock and arc_string_or_default helper for Option<Arc> Update From<&Node> and From<&PackageNode> implementations to clone Arcs instead of deep-copying data and to use arc_string_or_default for optional string fields Adjust tests and call sites to work with Arc-based fields instead of owned Vec/String values	modules/analysis/src/model.rs modules/analysis/src/model/graph.rs modules/analysis/src/model/roots.rs modules/analysis/src/service/test/mod.rs
Optimize Context::intern by using HashMap::entry with or_insert_with_key instead of separate contains/get and insert operations.	Replace manual contains_key/get/insert sequence with a single entry(s).or_insert_with_key call that stores an Arc of the key Return cloned Arc from the entry to ensure sharing while avoiding redundant lookups	modules/analysis/src/service/load/mod.rs
Minor API and test adjustments to match new shared data representations and removed imports.	Remove unused sbom_node_checksum import from analysis service module Update tests to compare against slices dereferenced from Arc<[Purl]> and Arc<[Cpe]> Use String::new() instead of "" literals for BaseSummary sbom_id/name/version/published in tests for consistency with new constructors	modules/analysis/src/service/mod.rs modules/analysis/src/service/test/mod.rs modules/analysis/src/model/roots.rs

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've left some high level feedback:

• The GraphQuery::Query optimization that gates nested purl/cpe materialization on query.q.contains("purl:") / "cpe:" is quite brittle; if the query syntax evolves (e.g., different casing, aliases, or structured fields), this detection will silently miss needed data, so consider delegating this decision to a parsed query structure instead of raw substring matching.
• The new ExternalSbomCache is keyed solely by node_id; if resolve_external_sbom behavior ever depends on additional context (e.g., SBOM ID, discriminator, or external type), this cache could return incorrect cross-request results, so it would be safer to make the key reflect all inputs that affect resolution.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `GraphQuery::Query` optimization that gates nested purl/cpe materialization on `query.q.contains("purl:")` / `"cpe:"` is quite brittle; if the query syntax evolves (e.g., different casing, aliases, or structured fields), this detection will silently miss needed data, so consider delegating this decision to a parsed query structure instead of raw substring matching.
- The new `ExternalSbomCache` is keyed solely by `node_id`; if `resolve_external_sbom` behavior ever depends on additional context (e.g., SBOM ID, discriminator, or external type), this cache could return incorrect cross-request results, so it would be safer to make the key reflect all inputs that affect resolution.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

Sourcery assessment

Approved.

[ctron]

Regarding the ExternalSbomCache concern: the cache key is node_id alone because resolve_external_sbom only takes node_id and a DB connection as inputs — there is no SBOM ID, discriminator, or external type parameter that could vary the result. The cache key already reflects all inputs that affect resolution.