Stop fetching every address on eth_getLogs when using RPC as a datasource

[ThallesP]

Envio fetches every single registered address when calling eth_getLogs:

While this approach makes sense during historical synchronization as it filters out unnecessary events from other addresses, it may not be optimal once we've reached the chain tip.

Issues with the current approach:

1. It produces numerous extra requests
2. It consumes substantial egress bandwidth, which is costly

I'm suggesting that when Envio reaches the chain tip, it should automatically switch to using eth_getLogs without any address filter, then internally discard logs from unregistered addresses.

I've created a proof of concept at notuslabs@643a719. I haven't tested it yet as I need to build and release it as a package for testing, but it could serve as inspiration (though it was generated with AI, so please take it with a grain of salt).

Additionally, it would be beneficial if we could configure this behavior via config.yml under the rpc section, allowing developers to set unfiltered addresses even during historical sync. The default setting should remain as live only.

[DZakh]

You can use https://docs.envio.dev/docs/HyperIndex/wildcard-indexing to stop sending the requests during backfill

[DZakh]

How many addresses do you index?

[ThallesP]

You can use https://docs.envio.dev/docs/HyperIndex/wildcard-indexing to stop sending the requests during backfill

makes sense, then ignore my suggestion for backfill (I do not need it but thought others would want it).

[ThallesP]

How many addresses do you index?

I register every Uniswap V3 pool, totaling approximately 37,000.

So in theory, I would perform 7.4 times more requests than simply using eth_getLogs without an address filter, which would also provide a significant speed improvement.

[DZakh]

Yeah, it makes sense. With HyperSync we have query caching layer in place, so it's not such a big problem, but definitely a huge egress cost with RPC :) We even almost got drained by AWS when in one of the old versions, a client indexed 2M of addresses 😅

[DZakh]

Even though, it's solved with HyperSync, we still want to have it improved with RPC. Additionally having a single request is much smaller latency than 2+

[ThallesP]

Yep, hopefully that will be resolved soon. In the meantime, I implemented a better workaround.

I built an RPC Proxy that cleanses the address parameter from eth_getLogs, forwards the request upstream, and then filters the response by address internally. It also caches the eth_getLogs response for approximately one minute to handle the additional seven requests.

So far, everything seems to be working perfectly. I'll maintain this solution until HyperIndex resolves the issue.

About HyperSync, that makes sense. We were previously using it, and after an update, the egress dropped massively.

[ThallesP]

In case someone needs it in the meanwhile, here's a gist: https://gist.github.com/ThallesP/10f558c53736f2b5de2f6a6676883923.