Fixes for Embers

[dylon]

This PR resolves COST_MISMATCH errors observed on observer nodes during block replay by implementing per-bind peek semantics, along with several supporting fixes for state hash consistency, exploratory deploy reliability, and replay correctness. It also transforms the RSpace from a globally-serialized tuple space into a concurrent architecture with per-channel parallelism (see docs/rspace/concurrent-rspace-architecture.md).

Changes

Core: Concurrent Par evaluation (rholang/src/rust/interpreter/reduce.rs)

• Concurrent evaluation with two-phase dispatch: Par branches evaluate concurrently via FuturesUnordered. When a COMM fires during Phase 1 (matching), body dispatch is deferred to a shared queue. After all matching completes, Phase 2 dispatches bodies sequentially — preventing the −644 COST_MISMATCH caused by body evaluation interleaving between RSpace (play) and ReplayRSpace (replay). Nested eval calls create their own Phase 1/Phase 2 cycle.
• Content-hash candidate ordering: Replaced thread_rng() shuffle with deterministic sorting by source.hash (Blake2b256). Cryptographic hashes provide uniform distribution (fair, no starvation) while being deterministic (consensus-safe). Required because random shuffle caused different COMMs on different runs under concurrent evaluation.
• locally_free clearing: Channels are normalized (locally_free cleared) before produce and consume operations. This fixes hash mismatches between the hot store (which uses Par::Eq/Hash, ignoring locally_free) and the history store (which uses bincode::serialize, including locally_free), ensuring channel identity is consistent across both stores.
• locally_free serde normalization: The protobuf-generated locally_free field is additionally normalized at the serialization layer via a custom serialize_with handler that always serializes as empty bytes, ensuring consistent Blake2b256 hashes regardless of stale bit-vector content.
• Stack size increase (.cargo/config.toml): RUST_MIN_STACK increased from 8MB to 32MB, and tokio worker thread stacks set to 32MB (node/src/main.rs), to accommodate deeper COMM cascades (50+ levels).

Core: Concurrent RSpace (rspace++, rholang)

• Atomic cost manager: Replace Arc<Mutex> with AtomicI64 + CAS loop, fixing a TOCTOU bug where two concurrent branches could both deduct past zero. Normalize consume-triggered COMM cost accounting to produce-triggered semantics, making gas costs order-independent.
• Hot store Mutex removal: Remove Mutex and Mutex wrappers from InMemHotStore. DashMap's per-shard RwLocks now serve their intended purpose.
• ISpace &self: Change all 12 &mut self methods in the ISpace trait to &self with interior mutability (Arc<Mutex<...>>, Arc<RwLock<...>>), enabling shared concurrent access without an outer Mutex.
• Per-channel-group locks: Add DashMap<u64, Arc<Mutex<()>>> for per-channel-group locking (Rust equivalent of Scala's TwoStepLock + MultiLock). Join patterns serialize through their group lock; single-channel operations use DashMap shard locks only. Deadlock prevention via sorted channel hash ordering.
• Remove interpreter lock: Remove Arc<tokio::sync::Mutex> from RhoISpace. Delete 19 .try_lock().unwrap() call sites. The ISpace is now accessed without any global lock.
• Sequential internal dispatch: continue_produce_process and continue_consume_process for persistent COMMs changed from join_all to sequential dispatch-then-re-produce/re-consume, ensuring deterministic ordering within a single COMM's lifecycle.

Core: Per-bind peek semantics (p_input_normalizer.rs, reduce.rs, rspace++)

• Per-bind peek tracking: Peek flag changed from a single boolean on Receive to per-bind tracking via BTreeSet. Each bind in a multi-bind receive can independently use peek (<<-), with data preserved only for peeked channels.
• ReceiveBind.peek field: Added peek: bool field to ReceiveBind protobuf message for per-bind peek propagation.
• RSpace peek semantics: Updated store_persistent_data and remove_matched_datum_and_join in both rspace.rs and replay_rspace.rs to respect the per-bind peeks set — peeked channels preserve their data, non-peeked channels have data consumed normally.

Core: Genesis pre-state hash handling (block_approver_protocol.rs, initializing.rs, interpreter_util.rs)

• Dynamic genesis hash: Replaced hardcoded empty_state_hash_fixed() constant with the genesis block's own pre_state_hash. The hardcoded constant breaks when evaluation order or trie hashing changes. The pre-state hash is now dynamically computed during compute_genesis() and stored on the RuntimeManager.
• genesis_pre_state_hash parameter: Added to compute_parents_post_state() so genesis validation uses the block's own hash rather than requiring the runtime manager to have computed genesis.
• InvalidPreStateHash replay failure: Blocks referencing non-existent state roots are now treated as invalid blocks (Right(None)) rather than internal errors (Left). UnknownRootError is detected immediately without retrying (it's deterministic).

Core: spawn_runtime_at() (runtime_manager.rs)

• New method: spawn_runtime_at(hash) creates an RSpace child directly at a target state hash, avoiding the bug where spawn() + reset() leaves the history reader stale (history repository is updated but the hot store's reader still points to the old root).
• Registry init flag: spawn_runtime and spawn_replay_runtime now pass init_registry=false since the registry is already in the state from genesis — re-initializing per block would corrupt state.

Core: Exploratory deploy fix (runtime.rs)

• Fresh key pair per call: Exploratory deploy now generates a fresh secp256k1 key pair and wall-clock timestamp for each invocation, ensuring the deploy's RNG seed is unique and the externally-computed return channel always matches the ret channel created inside eval_new.

API: Trie stats endpoint (web_api.rs, shared_handlers.rs)

• GET /trie-stats: New LFS diagnostic endpoint that returns block number, state hash, and read-only status for comparing validator and observer state completeness.

API: Block propose retry (block_api.rs)

• Retry on "already in progress": When a propose is already in progress, the system now retries with configurable delay instead of silently returning.

Infrastructure: macOS LMDB semaphore cleanup (resources.rs)

• atexit handler: On macOS, LMDB uses POSIX named semaphores that are never cleaned up when lazy_static values aren't dropped. Added startup cleanup of orphaned semaphores and an atexit handler to prevent ENOSPC after repeated test runs.

Diagnostics

• Extensive structured tracing added throughout the evaluation pipeline: RNG split behavior, COMM firing details, channel hashing, hot store pending state, registry probes, cost trace sequencing, checkpoint roots, and exploratory deploy flow.

Test coverage

• Concurrent COMM: Validates COMM fires correctly when send and receive coexist in a single Par under concurrent evaluation.
• Genesis pre-state hash replay: Verifies replay_compute_state succeeds using the block's own pre_state_hash. Uses with_isolated_runtime_manager for LMDB state isolation.
• spawn_runtime_at consistency: Confirms data written by a deploy is readable at the new state but not at the old state.
• Mixed peek multi-bind: Validates per-bind peek preserves peeked channel data while consuming non-peeked channel data.
• locally_free clearing: Verifies COMM fires despite stale locally_free bits on substituted channels.
• Exploratory deploy integration test: End-to-end test deploying a persistent contract, finalizing, then invoking via exploratory deploy.
• Updated expected hashes and costs: Runtime spec hash updated; empty_state_hash test changed to verify non-trivial hash rather than hardcoded comparison; cost accounting values updated for receives-first evaluation and per-bind peek semantics.
• RSpace peek property tests: New proptest-based tests validating peek data preservation, non-peek data removal, and mixed peek/non-peek selective removal across produce-then-consume and consume-then-produce orderings.
• Stress tested: 5/5 passes on full util::rholang suite (39 tests per run, zero failures) with concurrent evaluation.