DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs

[aparajit-pratap]

Purpose

Continued from PR #16255 : Improve Library Loading of many DYFs.

Some metrics (3 tries):

Customnode loading performance (clockwork package = ~453 custom nodes):
Deserialize cache-file when empty - ~0 ms
Deserialize cached file - 117/39/48 ms
Construction of nodes from cache - 175/105/89 ms
Construction from JSON - 1000/368/376 ms
Serialize cache to file - 31/20/1 ms

It can be seen that in general using the cache (deserialization of cached file + construction of nodes from in-memory cache) takes less than half the time it would take to construct the nodes from JSON deserialization (and that is after the optimizations in this PR for JSON deserialization).

Declarations

Check these if you believe they are true

• Is documented according to the standards
• The level of testing this PR includes is appropriate
• User facing strings, if any, are extracted into *.resx files
• Snapshot of UI changes, if any.
• Changes to the API follow Semantic Versioning and are documented in the API Changes document.
• This PR modifies some build requirements and the readme is updated
• This PR contains no files larger than 50 MB
• This PR introduces new feature code involve network connecting and is tested with no-network mode.

Release Notes

(FILL ME IN) Brief description of the fix / enhancement. Use N/A to indicate that the changes in this pull request do not apply to Release Notes. Mandatory section

Reviewers

(FILL ME IN) Reviewer 1 (If possible, assign the Reviewer for the PR)

(FILL ME IN, optional) Any additional notes to reviewers or testers.

FYIs

(FILL ME IN, Optional) Names of anyone else you wish to be notified of

[github-actions]

See the ticket for this pull request: https://jira.autodesk.com/browse/DYN-9011

[saintentropy]

In terms of the pruning of cache items that represent files that are no longer on disk or no longer match the hash maybe something like this:

public void Prune()
{
    foreach (var key in cache.Keys.ToList())
    {
        //get the path from the key
        int idx = key.IndexOf(".dyf", StringComparison.OrdinalIgnoreCase);
        string path = idx >= 0 ? key.Substring(0, idx + 4) : string.Empty;

        if (File.Exists(path))
        {
            //Check the files write time and length (same mechanism as original key)
            var fileinfo = new FileInfo(path);
            var lastWriteTime = fileinfo.LastWriteTimeUtc;
            var length = fileinfo.Length;
            var newKey = path + lastWriteTime.ToString() + length.ToString();
            if(newKey != key)
            {
                //If keys do not match then the file has been updated, remove from cache
                cache.Remove(key);
            }
        }
        else
        {
            //file no longer exists, remove from cache
            cache.Remove(key);
        }
    }
}

Not sure exactly when to tigger. We could do it on exit but that might take a while. Maybe a background task? Again not sure when to tigger but maybe only needs to be done once per user session

[aparajit-pratap]

This passes here except for one test (which I think is unrelated and failing due to it being flaky)

[Copilot]

Pull Request Overview

This PR enhances library loading performance by caching custom node metadata, adds a fallback error message resource, and refactors related parsing logic.

• Introduced JsonCache<T> to store and serialize CustomNodeInfo on disk with stale‐entry cleanup.
• Updated TryGetInfoFromPath to use the cache and unified JSON/XML parsing paths.
• Added a new resource string for unknown file‐processing errors and extended CustomNodeDefinition to support JSON‐based parsing.

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/DynamoCore/Properties/Resources.resx	Added UnknownErrorProcessingFile resource for fallback error messaging.
src/DynamoCore/Properties/Resources.en-US.resx	Added US variant for the unknown error resource.
src/DynamoCore/Models/DynamoModel.cs	Removed existing‐element sync logic in InitializeCustomNodeManager and simplified early return.
src/DynamoCore/Core/CustomNodeManager.cs	Added JsonCache<T> class, initialized cache on startup, and offloaded stale‐entry removal.
src/DynamoCore/Core/CustomNodeDefinition.cs	Added manual JSON parsing (GetFromJsonDocument) and [JsonConstructor] for CustomNodeInfo.

Files not reviewed (1)

• src/DynamoCore/Properties/Resources.Designer.cs: Language not supported

Comments suppressed due to low confidence (4)

src/DynamoCore/Core/CustomNodeManager.cs:44

• New caching logic in JsonCache<T> and its interaction with disk I/O and stale‐entry removal require unit tests to verify correct serialization, deserialization, and cleanup behavior.

        private class JsonCache<T>

src/DynamoCore/Models/DynamoModel.cs:1517

• The removal of the block that synced existing CustomNodeSearchElement instances (and updated them) means updates to already‐loaded nodes are no longer applied. Consider restoring that logic to keep the search model in sync when a node is reloaded.

                    return;

src/DynamoCore/Core/CustomNodeManager.cs:861

• Variable ex is declared inside nested scopes and is out of scope here, causing a compile error. Declare Exception ex; before the try block or consolidate error handling to ensure ex is available at this point.

                if (ex == null)

src/DynamoCore/Core/CustomNodeDefinition.cs:346

• [nitpick] Changing IsVisibleInDynamoLibrary from a read‐only property to a public setter may break invariants. If external mutation is unintended, revert to a private setter or validate assignments.

        public bool IsVisibleInDynamoLibrary { get; set; }

[mjkkirschner]

hey @aparajit-pratap - for the dynamo as a service case - I don't think caching will help us (at least not with our single use architecture) and writing to disk will likely be a waste of time for us.

Two questions with that context:

1. Can you characterize how much overhead generating and saving the cache adds?
2. Is there a startup configuration or some other mechanism to control if a cache is produced/saved?

[aparajit-pratap]

ions with that context:

1. Can you characterize how much overhead generating and saving the cache adds?
2. Is there a startup configuration or some other mechanism to control if a cache is produced/saved?

@mjkkirschner, thanks for the review. I'm trying to measure the overall performance gain (if any) with this PR; will share the results shortly. I haven't characterized the overhead related to saving the cache, no.

No, there's currently no startup config to control if the cache is generated but I could add one. I can see why for DAAS it might not be that useful, as with the use-and-throw containers, you won't be reloading the same packages and dyf's repeatedly.

[reddyashish]

Changes look good.