[IMD] Initial Market Discovery

[mojoX911]

Problem statement

Currently, the market is only available via blockchain. Scanning the blockchain, even with a network birthdate, with multiple threads, will still take some time. For mainnet, it can potentially be in hours. It's similar to a node IBD process. Named here as IMD.

This process has to be optimized maximally at the core lib routines for the fastest market scanning. Alternatively a parallel marketplace can be seeded in Nostr for faster background IMD. Later on, ideas similar to utreexos can be adopted for even faster IMD at mobile.

But for now, the core lib has to be optimized maximally at the core lib level.

Basic optimizations.

• start with a static const start date, NetworkBirthDate, and always sync from this instead of genesis.
• Make this configurable in the taker and maker configs. We will release with a sane default. Not needed at app layers yet.
• Run parallel threads and do parallel processing. Break the chain into n parts. Each part is scanned by an individual thread and a list of candidate transactions (that match the fidelity pattern) is sent to another thread pool for connection_checkers.
• Two thread pools can be effectively used to parallelize two major categories of discovery work. Scanning for candidate transactions in blocks and checking for liveliness for each candidate.
• Only after passing through connection checkings, the Tor addresses get added to good makers. For failed connections, they get added to bad makers. Or better add a new category in the offerbook unresponsive makers. Not all bad makers are unresponsive.
• After each IMD update, the address book in the disc. Maintain a record of all categories.
• Check from the already known unresponsive makers list to skip avoidable failures. At connection check, quickly match if it was known as unresponsive; if yes, skip trying the connection check early. Tor connections are expensive and take a lot of time to fail, too.

Advanced

• Other small corners of efficiency that we can harness not listed above.
• More advanced ideas can be added later, such as merkleizing the market data (utreexo), alternate seeding via Nostr, etc. Those can be tackled v0.2.1 onwards.

[stark-3k]

Another optimization could be to use Compact Block Filters to entirely skip the blocks that do not contain any Tor addresses while doing the IMD which will help us skip 99% of unnecessary scanning.

[Wukong247]

Another optimization could be to use Compact Block Filters to entirely skip the blocks that do not contain any Tor addresses while doing the IMD which will help us skip 99% of unnecessary scanning.

How would you know the exact scriptPubKey's? CBF doesn't do prefix matching.

[stark-3k]

Another optimization could be to use Compact Block Filters to entirely skip the blocks that do not contain any Tor addresses while doing the IMD which will help us skip 99% of unnecessary scanning.

How would you know the exact scriptPubKey's? CBF doesn't do prefix matching.

I was thinking including an output above dust threshold in the fidelity transaction to a known burner scriptPubKey for all makers. Not a very clean solution but will optimize the IMD quite a bit.

[mojoX911]

There are multiple potential approaches that we can take. Maybe even combining multiple solutions together.

I would suggest targeting the basic optimizations that we can do in our code for now. Then design approaches for more advanced solutions like CBF, Utreexo, later.

[mojoX911]

As discussed in the dev call.

• Start sync from the tip instead of the start date. Sync up to the last 6 months from the current tip.
• Improve the market management in the offerbook structure. good makers, unresponsive makers, bad makers. Don't connect to known unresponsive makers in IMD.
• Multi threaded IMD.
• Check for fidelity spend in both IMD and ZMQ discovery.