crafterDB.specializationData stores ~99% redundant static data, causing extreme memory usage with multiple alts

[Sponsorn]

Summary

The crafterDB serializes full SpecializationData objects per-recipe per-character, including static fields (node names, icons, perk thresholds, percentDivisionFactor, zero-value stats) that are already available in SPECIALIZATION_DATA.NODE_DATA. With many crafting alts, this causes SavedVariables to balloon massively.

My setup: 22 characters, 15 with active crafting professions

• CraftSim.lua SavedVariables: 107 MB on disk (~450+ MB in Lua memory)
• crafterDB.specializationData alone: 5.7 million lines (94% of the file)
• Single character example (Eng + LW): 267 recipe entries x ~2,500 lines each = 698,730 lines

Root Cause

NodeData:Serialize() (NodeData.lua:220-235) stores 9 fields per node:
nodeID, name, icon, rank, maxRank, active, professionStats, maxProfessionStats, perkData[]

Each professionStats contains 6 stat objects with name, value, extraValues, percentDivisionFactor — even when 5 of the 6 are zeroes.

Of these, only rank is character-specific. Everything else is:

• Static Blizzard data lookupable by nodeID (name, icon, maxRank)
• Derivable from rank (active, professionStats, maxProfessionStats)
• Constants (percentDivisionFactor, stat name strings)
• Already shipped with CraftSim (perkData thresholds and stats in SPECIALIZATION_DATA.NODE_DATA)

Additionally, data is stored per-recipe, but recipes in the same profession share the same spec tree. The same node ranks are duplicated across every recipe that references them.

Proposed Fix

Phase 1: Compact serialization (minimal refactor)

Change NodeData:Serialize() to only store what's dynamic:

function CraftSim.NodeData:Serialize()
    return { nodeID = self.nodeID, rank = self.rank }
end

Update NodeData:Deserialize() to reconstruct from static data + rank — the constructor already does this for the current character, so it's reusing the same path.

SpecializationData:Deserialize() would call UpdateProfessionStats() after rebuilding nodes (this method already exists).

Files changed: ~3 (NodeData.lua, SpecializationData.lua, crafterDB.lua for migration)
Lines changed: ~100-150 + ~50 for migration

Phase 2: Per-profession storage (optional, bigger win)

Change storage key from per-recipe to per-profession:

-- Before: 267 duplicate entries
crafterDB.data[crafter].specializationData[recipeID] = { ... }

-- After: one entry per profession
crafterDB.data[crafter].nodeRanks[professionKey] = {
    [nodeID1] = 5,
    [nodeID2] = 3,
    ...
}

GetSpecializationData() would use the existing recipeMapping to filter relevant nodes for each recipe.

Expected impact

	Lines per character (Eng+LW)	Disk size (22 chars)
Current	698,730	~107 MB
Phase 1	~3,500	~1 MB
Phase 2	~50	~22 KB

Lua memory savings would be even larger due to table/string overhead per entry.

Environment

• Retail WoW 12.x
• CraftSim latest
• 22 characters, 15 with active crafting professions across 3 expansions

[derfloh205]

thats a pretty good find!
I will see that I can implement those optimizations asap