Improve json parsing performance

[CptMoore]

TLDR: Too much effort, therefore postponed

currently BT loads jsons

1. BT request resources via datamanager
2. datamanager loads json as string
3. target class converts string to nested dictionary<string, object>
4. helper utility then maps dictionary to target object via reflection

• there is some lookup-caching but its still meta-code working via reflection apis

an improvement could look like

1. BT requests resources
2. maps a json file (bytes) to target object
   • use cached IL-processing/mapping code (see msgpack which partially does that)
     • removes costly meta-processing (if/else branches etc..)
   • parse bytes with help of sse or avx512 simd
     • simdjon is a very fast parser. Issues:
       • C# <-> native overhead is large
       • the .NET only solution is not for .NET Framework, would require backport
         • span etc.. is already backported, no issue
         • missing SIMD support in the mono jit employed by Unity
       • is only the parser, not the mapper
       • fix: probably have to implement ourselves just for BT' .NET version
   • never parsing to a string unless a string is actually needed
     • json is utf8, while .NET strings are utf16, conversion always costs
     • enums etc.. could be cached to their utf8 byte[] representation
   • the fastJson parser used in vanilla code and custom mapper by HBS have some deterministic but undefined behavior regarding order of field/property initialization. One would need to replicate the same order as vanilla to avoid NREs or other logic errors.

other ideas:

• use newtonsoft json not only for merging but also datamanager loading
  • newton is much (3-6x) slower than the fastJson+custom mapper solution
  • newton is also not fully compatible, order still has to be fixed but otherwise this branch has much solved already (custom converters, settings etc..): https://github.com/BattletechModders/ModTek/tree/working-on-newtonsoft-compatibility
• msgpack would still be fastest, but that would only work for a cache, as modders cant use notepad to modify data
  • see cmission/kmission's CustomPrewarm as an example on how to cache
    • much is hardcoded, could be made generic with some effort in msgpack setup for resolver+formatters (and CC+ME being msgpack compatible); could then be integrated into modtek