perf: bulk DB writes and reads during indexing

[bonustrack]

Summary

Cuts DB round-trips during indexing by batching writes and (opt-in) reads against Postgres. Ported from the approach Envio HyperIndex uses for its historical sync.

Three stacked commits:

• 84d058c — Per-block write buffer. All entity inserts/updates/deletes, _checkpoints, _blocks, and _metadatas.last_indexed_block for a block accumulate in a new EntityWriteBuffer and are flushed inside one knex.transaction at the end of the block. Per touched table: one bulk UPDATE ... WHERE id IN (...) to close previous ranges + one bulk INSERT. Replaces the per-entity transaction that Model.save() used to open.
• c7c3327 — Preload phase. Each event entry in the source config can declare preload_fn pointing to a Preloader function. Before handlers run for a block, all matched preloaders run in parallel and return { table, ids }[]. The framework issues one SELECT ... WHERE id IN (...) per table to warm the buffer cache, so subsequent Model.loadEntity calls in handlers are free. Preloaders are registered alongside writers on the indexer (second constructor arg, or options.preloaders for HyperSyncEvmIndexer).
• 1be74e7 — Opt-in cross-block batching. New batch_size config field (default 1, i.e. today's exact semantics). When >1 during historical sync, buffer.flush is called every N blocks instead of every block; within a batch, an entity's intermediate states collapse (the new row's block_range lower bound and the closed range's upper bound both use the block where the entity was first modified in the batch). Pre-batch history is preserved exactly. Near tip (blockNumber >= preloadEndBlock), batching is forced to 1 so reorgs and live queries stay correct.

Effect on DB traffic

For a block with N entity ops:

• Before: ~2N + 3 DB round-trips, ~N independent transactions.
• Phase 1 (batch_size=1, no preload): ~5 round-trips, 1 transaction per block.
• Phase 2 (batch_size=1, preload declared): ~5 writes + 1 bulk SELECT per touched table per block.
• Phase 3 (batch_size=K): same per-batch shape amortized over K blocks.

Handler failures now roll the whole block back atomically rather than leaving partial state.

Correctness notes

• Mid-batch errors discard the buffer and rewind blockNumber to the batch's first block so no committed work is skipped.
• handleReorg walks back from the DB's last_indexed_block rather than the in-flight block number, so uncommitted batch blocks can't be mistakenly treated as the last good block.
• Container.getBlockHash consults pending block hashes in the buffer first, so reorg detection works mid-batch.
• batch_size=1 (default) produces byte-identical DB state to today.
• HyperSync inherits the preload phase from EvmProvider without changes.

Test plan

• yarn build passes
• yarn test — all 68 existing tests pass
• yarn lint passes
• Run an existing examples/ indexer with batch_size=1 on this branch and master; diff final DB state (rows, block_range, _checkpoints, _blocks, _metadatas) — expect identical output.
• Benchmark on an Arbitrum contract (~100k blocks of a busy ERC-20) master vs branch; target ≥5× fewer DB round-trips per block. ≥15× wall-time speedup with batch_size=50 + preload.
• Manual smoke: run with batch_size=10, assert GraphQL historical queries (entity(block: N)) return the final batch state for any N inside the batch range, and the pre-batch state for N before it.
• Force a reorg mid-batch and confirm no stray rows remain.

🤖 Generated with Claude Code