Improve market state retrieval

[LesnyRumcajs]

Summary

It turns out the Filecoin.StateMarketDeals can't be run on the mainnet without knocking out a decent host (64 GiB). This needs to be improved, either by optimising the current method implementation or proposing something else. The current one's logic seems to be hardly usable—it creates a 100 million-entry map in memory, iterates over proposals to fill it, and then sends the response (yes, the 100M entries map) via JSON-RPC.

The first challenge is the proposal's AMT iteration, which knocks out Forest due to caching. This can potentially be mitigated by filecoin-project/ref-fvm#2189.

The second challenge is the response on its own. We'd probably be better off with some streaming solution for this amount of data.

Completion Criteria

• Investigate what can be done to facilitate market state dumping,
• If feasible, implement some PoCs

Additional Links & Resources