EVM: reconcile_tx_nonce_state finalizes mined transactions as Failed on transient empty receipt (v1.5.0 regression)

[Kartik4152]

Summary

On EVM networks, reconcile_tx_nonce_state (added in #726, shipped in v1.5.0) can finalize a successfully mined transaction as Failed when get_transaction_receipt transiently returns Ok(None) for a hash that is actually on-chain. The transaction is recorded with:

Nonce <N> consumed externally (on-chain nonce: <N+1>). No matching transaction hash found on-chain.

…even though the relayer's own tracked hash mined at exactly nonce N with a status = 0x1 receipt. Because the transaction is finalized to Failed, no confirmed notification is emitted for a transaction that actually succeeded — a silent success-reported-as-failure.

This is a regression from v1.4.0, which handles the same situation correctly.

Impact

• Successfully-mined transactions are irreversibly marked Failed.
• Webhook/notification consumers never learn the transaction succeeded, while the on-chain state change did occur.
• The trigger is ordinary RPC read-consistency lag on load-balanced providers, so healthy deployments are affected intermittently. In one incident we observed 26 consecutive fast transactions on Base finalized this way during a ~45-minute provider lag window — every one was mined with a status = 0x1 receipt at the exact hash and nonce the relayer had tracked.

Affected versions

v1.5.0 and current main. v1.4.0 is not affected (see Regression).

Root cause

src/domain/transaction/evm/status.rs — reconcile_tx_nonce_state (v1.5.0):

Step 1 checks the current hash's receipt (status.rs:671):

match self.provider().get_transaction_receipt(hash).await {
    Ok(Some(_)) => return Ok(None),    // found → defer to normal flow
    Ok(None)    => { /* continue */ }   // ← treated as authoritative "not mined"
    Err(e)      => { had_rpc_errors = true; }
}

Step 3 only guards against Err (status.rs:722, :748):

if had_rpc_errors { return Ok(None); }        // defers on transient RPC error
...
if on_chain_nonce > tx_nonce {
    // "Nonce {tx_nonce} consumed externally ... No matching transaction hash found on-chain."
    // → update_transaction_status(..., Failed, ...)   (irreversible)
}

The code deliberately avoids the irreversible determination when a receipt call returns Err. But a stale Ok(None) is also an unreliable signal: on load-balanced / eventually-consistent RPC endpoints, eth_getTransactionReceipt can return null for a transaction that is already mined, while eth_getTransactionCount (served by a more up-to-date node) already reflects the advanced nonce. The two reads are not guaranteed mutually consistent within a single pass.

So for the relayer's own mined transaction, both can be true in the same reconcile pass:

• get_transaction_receipt(H) → Ok(None) (lagging node)
• get_transaction_count(addr) → > tx_nonce (up-to-date node)

→ the code concludes "consumed externally" and finalizes Failed, even though H is on-chain.

Trigger

The reconcile pass runs via the one-shot tx_nonce_reconcile_trigger hint, set when a resubmission is rejected with nonce too low / already known. On fast networks this is common and benign — the tx already mined, so the resubmit is correctly rejected — but that rejection now routes into reconcile, which mis-finalizes it on a stale receipt read.

Regression from v1.4.0

v1.4.0 has no reconcile_tx_nonce_state. It handles the same nonce too low / already known resubmission error defensively — src/domain/transaction/evm/evm_transaction.rs (v1.4.0):

// SAFETY CHECK: If we get "already known" or "nonce too low" errors,
// it means a transaction with this nonce was already submitted
if is_already_submitted {
    warn!(... "resubmission indicates transaction already in mempool/mined - keeping original hash");
    true
}
...
// Keep original hash and data - just ensure status is Submitted
TransactionUpdateRequest { status: Some(TransactionStatus::Submitted), .. }

It keeps the transaction Submitted and keeps polling; once the receipt is visible it transitions to Confirmed and the notification fires. A stale Ok(None) only delays confirmation; it never fails it. #726 replaced this tolerant behavior with eager finalization.

Steps to reproduce

1. EVM network with fast blocks and a load-balanced RPC endpoint (e.g. Base).
2. Submit fast transactions steadily so some enter the resubmission window before their receipt is observed.
3. During a period where eth_getTransactionReceipt lags eth_getTransactionCount (normal under provider load / node desync), observe transactions finalized Failed with reason Nonce <N> consumed externally ... No matching transaction hash found on-chain.
4. Confirm via eth_getTransactionByHash / eth_getTransactionReceipt that the tracked hash is a mined tx with a status = 0x1 receipt at nonce N.

Suggested fix

Do not treat a single Ok(None) receipt read as authoritative before the irreversible finalization. Options, roughly in order of robustness:

1. Disambiguate via the hash directly: before finalizing, call eth_getTransactionByHash for each known hash; if any returns a transaction with a non-null blockNumber, the nonce was consumed by this relayer's transaction (not externally) → defer to normal flow so it confirms.
2. Debounce: require the receipt == None + on_chain_nonce > tx_nonce condition to persist across ≥2 consecutive reconcile passes (with a short delay) before finalizing, since the condition is self-healing.
3. Retry with backoff on get_transaction_receipt for known hashes before concluding absence.
4. Minimal change: treat Ok(None) in step 1 like Err (defer) when it co-occurs with an advanced on-chain nonce — a mined-but-not-yet-visible receipt is indistinguishable from external consumption on a single read, so it should not drive an irreversible Failed.

Happy to open a PR if maintainers agree on the preferred approach.