feat: Add stateless-executor guest program and SP1 cluster infrastructure#15
Open
WiseMrMusa wants to merge 34 commits intomasterfrom
Open
feat: Add stateless-executor guest program and SP1 cluster infrastructure#15WiseMrMusa wants to merge 34 commits intomasterfrom
WiseMrMusa wants to merge 34 commits intomasterfrom
Conversation
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Han <tinghan0110@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
…ion 4bac87e for ere packages in Cargo.toml and Cargo.lock
- Introduced Cluster variant in Resource enum for SP1 cluster resources. - Updated ProverResourceType conversion to handle Cluster using default ClusterProverConfig.
- Introduced a new `docker-compose.yml` file for managing SP1 Cluster services including Redis, PostgreSQL, API, Coordinator, CPU, and GPU nodes. - Added `start-sp1-cluster.sh` script for starting the cluster with options for GPU nodes and mixed worker modes. - Created `stop-sp1-cluster.sh` script for stopping the cluster and managing persistent data and images. - Enhanced user experience with logging and help messages in the scripts.
- Added validation in the CLI to ensure that the Cluster resource can only be used with SP1 zkVMs. - Introduced `bail` from `anyhow` for error handling in case of invalid resource configurations.
Adds a new crate that provides pure EVM transaction execution without pre-execution validation or post-execution consensus checks. Key components: - stateless_execution_with_trie: Core execution function returning bool - WitnessDatabase: EVM database backed by StatelessTrie witness data - RethStatelessExecutorGuest: Guest implementation for zkVMs - SP1 and RISC0 entry points for zkVM compilation This enables accurate benchmarking of raw EVM execution cycles in zkVMs by skipping all validation overhead.
Updates both root and ere-guests workspace Cargo.toml to include: - reth-stateless-executor as a workspace member - Required reth dependencies (reth-evm, reth-revm) - Dependency declarations for the new crate
Adds stateless_executor module to benchmark-runner for execution-only benchmarking support: - stateless_executor/reth.rs: Input preparation for Reth executor - Updated lib.rs to export the new module - Made BlockMetadata.block_used_gas public for reuse
…-executor Adds zkVM-specific entry points for all supported platforms: - Airbender: With custom allocator and runtime - OpenVM: With crypto provider installation - Pico: With KZG proof verification crypto provider - ZisK: Standard entry point These mirror the structure of stateless-validator entry points.
Introduces a new stateless_executor module with the following features: - Added stateless_executor.rs for handling execution client variants and input preparation. - Updated reth.rs to utilize StatelessExecutorFixture for input processing. - Enhanced read_benchmark_fixtures_folder to read and parse benchmark fixtures. This lays the groundwork for execution-only benchmarking of stateless programs.
…cutor and validator to allow both input_file and input_folder This commit introduces the ability to specify either an input folder or a single input file for the stateless executor and validator. The `stateless_executor_inputs` and `stateless_validator_inputs` functions are updated to support this new functionality, along with corresponding changes in the CLI to accept an optional input file argument. This improves flexibility in benchmarking by allowing users to work with individual benchmark fixture files directly.
…r stateless executor This commit modifies the CLI to set a default value for the execution client argument, allowing for easier configuration. Additionally, the guest relative path for the Reth execution client is simplified by removing the specific subdirectory, enhancing the path resolution for the stateless executor.
This commit introduces a new `MemoryTracker` struct to monitor memory usage during the proving process. It tracks initial, peak, and average memory usage, and integrates memory sampling into the benchmark runner. The proving metrics now include memory usage statistics, enhancing performance analysis capabilities.
…d runner.rs This commit introduces optional memory tracking capabilities in the benchmark runner. The `Cargo.toml` is updated to include new features for memory tracking, and the `runner.rs` file is modified to conditionally compile memory tracking logic based on the feature flag. This enhances the ability to monitor memory usage during the proving process, aligning with previous enhancements to performance analysis.
…chmark CLI This commit introduces a new `Network` resource type in the CLI, allowing users to specify network proving. It also adds validation to ensure that network proving is only used with SP1, enhancing the robustness of the command line interface for the zkVM benchmarker. The changes improve user experience by preventing misconfigurations related to resource selection.
… in reth.rs This commit updates the input handling in the `stateless_executor_inputs_from_fixtures` function to pass a reference to the benchmark wrapper instead of moving it. This change ensures that the benchmark wrapper can be reused, improving memory efficiency and preventing potential ownership issues.
This commit cleans up the `runner.rs` file by removing the unused `PublicValues` import from the `ere_zkvm_interface` module. This enhances code clarity and maintains a cleaner codebase.
…nd version 4bac87e for ere packages in Cargo.toml and Cargo.lock
Regenerate lockfile to resolve Git merge conflict markers that were preventing the project from building. The conflicts were between different versions of ere dependencies.
… commit hashes This commit modifies the Rust toolchain version in multiple workflow files to use specific commit hashes instead of the stable tag. This change ensures consistency in the toolchain used across different CI jobs, enhancing build reliability.
…e files This commit refactors several files by reorganizing and cleaning up import statements for better readability and consistency. Additionally, it enhances code formatting in the `runner.rs`, `stateless_executor.rs`, `stateless_validator.rs`, and other related files, ensuring a more maintainable codebase. These changes do not alter functionality but improve overall code clarity.
This commit updates multiple GitHub workflow files to include the `toolchain` parameter when installing the Rust toolchain. This change ensures that the toolchain is explicitly set to either stable or nightly, improving consistency and clarity in the CI configuration.
This commit introduces a new `Cluster` resource type in the CLI, allowing users to specify cluster proving. It also updates the validation logic to ensure that both network and cluster proving are only used with SP1, improving the robustness of the command line interface for the zkVM benchmarker. Additionally, the `MemoryTracker` struct is enhanced with a default implementation and methods for peak and initial memory usage are marked as `const`, optimizing performance.
This commit enhances the `proving` structure in the metrics README by adding fields for peak, average, and initial memory usage in bytes. These additions improve the granularity of memory tracking during the proving process, aligning with recent enhancements in memory tracking capabilities.
This commit refines the header parsing logic in the `RpcBlocksAndWitnessesBuilder` by trimming whitespace and filtering out empty headers. This enhancement ensures that only valid headers are processed, improving the robustness of the RPC request handling.
590a0d6 to
97b6421
Compare
WiseMrMusa
pushed a commit
that referenced
this pull request
Jan 8, 2026
Create benchmark runner and allow parallel execution
97b6421 to
590a0d6
Compare
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
e4bdeab to
33861da
Compare
This commit updates the versions of several dependencies in `Cargo.lock` and modifies the source revisions for the `ere` packages in `Cargo.toml` and `ere-guests/Cargo.toml`. The changes include downgrading `itertools`, `syn`, and `windows-sys` versions, as well as updating the source revision for various `ere` packages to a more recent commit. These updates aim to ensure compatibility and stability across the project.
33861da to
817c03f
Compare
This commit introduces several new workspace dependencies in `Cargo.toml`, including `openvm-k256`, `openvm-keccak256`, `openvm-kzg`, `openvm-p256`, `openvm-pairing`, and `openvm-sha2`, enhancing the functionality of the OpenVM platform. Additionally, a new module for `openvm_revm_crypto` is added in `main.rs`, setting the stage for further development in cryptographic functionalities.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces the stateless-executor guest program, a new benchmarking workload that executes Ethereum blocks without pre/post execution validation overhead. It also adds initial SP1 cluster Docker infrastructure for local proving.
Detailed Changes
1. New Stateless Executor Guest Program
Added a complete
stateless-executorcrate underere-guests/that provides execution-only benchmarking capabilities:Core Library (
ere-guests/stateless-executor/src/)lib.rsexecution.rsguest.rswitness_db.rszkVM Entry Points
Each supported zkVM has its own entry point:
risc0/src/main.rssp1/src/main.rsopenvm/src/main.rspico/src/main.rszisk/src/main.rsairbender/src/main.rs2. Benchmark Runner Integration
Updated
crates/benchmark-runner/to support the stateless-executor:stateless_executor.rsstateless_executor/reth.rsrunner.rsCargo.toml3. CLI Enhancements
Updated
crates/ere-hosts/src/cli.rs:--guestflag forstateless-executorvsstateless-validator--input-fileand--input-foldernetworkas valid resource type for SP1 zkVM4. Memory Tracking (
crates/metrics/)Added memory tracking capabilities:
5. SP1 Cluster Infrastructure
Added Docker-based SP1 cluster setup under
scripts/sp1-cluster/:docker-compose.ymlstart-sp1-cluster.shstop-sp1-cluster.shServices included:
6. Dependency Updates
Updated
Cargo.tomlandCargo.lock:NethermindEth/ererepository4bac87e7. Cluster Resource Support
Added cluster resource type for SP1 proving:
Why These Changes?
Stateless Executor Benefits
SP1 Cluster Benefits
Testing
Breaking Changes
None. This PR adds new functionality without modifying existing APIs.