[EVM] Backup reth state from the sequencer periodically in some database

[Manav-Aggarwal]

We'd like to support periodic backups for sequencer's reth state in some database in the event of state loss on the sequencer side.

This is a must-have for production-environments.

It's likely that reth already has something like this that can be enabled via a flag, but it needs to explored and documented well in Rollkit docs.

[auricom]

Reth Current State

I didn't find a built-in feature for exporting state on reth as of now ? While Reth supports importing dumps created by other execution clients (like Geth), I did not find trace of the reverse export functionality.

ERA Format Support (In Progress)

The Reth team is actively developing ERA-encoded import/export feature: Core logic is merged (PR #15909) ; however there is no command-line interface yet.
Tracking issues: #16899 and #16892

Alternative Approaches

Custom RPC-based Export Script

• Attempted a Python script to read blocks from genesis via RPC endpoint, importing web3 and rlb librairies
• Custom logic to encodes block in a RLP format compatible with Reth
• Verdict: Not recommended due to safety concerns compared to native implementation

Datadir Backup Method

• Run a full node, sync it to latest block
• Shut down the node gracefully
• Create a backup of the entire datadir
• Verdict: Safe and reliable, but require specific deployment orchestration

[WalidOfNow]

@auricom The feature we are asking for here is about the ability of the sequencer node to sync up reth state from onf the full nodes. The scenario as following:

1- Full node A and B are running and are in sync with sequencer node.
2- Sequencer node crashes and states is erased.
3- Sequencer goes back online and sees that there's no state, goes and syncup the state from node A or B.
4- Sequencer verfies that this state matches the latest state written to the DA (optional? not sure how doable is this)
5- Everything goes back to normal and sequencer starts producing new blocks and gossip it to the full nodes.

[auricom]

Here's a procedure for automating reth state backup rollkit/docs#582

However this wouldn't cover a sequencer state restoration scenario (that would likely rely on external snapshots as p2p sync could take a long time)