Redis workers reuse Apalis WorkerId across replicas, preventing orphan recovery during rolling deployments

[snnbotchway]

Describe the bug

All Redis-backed Apalis workers use fixed names, such as
transaction_status_checker_evm:

https://github.com/OpenZeppelin/openzeppelin-relayer/blob/main/src/queues/redis/worker.rs

Apalis uses this name as the Redis consumer identity and as part of the key for
the worker's in-flight set. When two Relayer replicas overlap, as they normally
do during a rolling deployment, both processes therefore share one Redis worker
identity.

If the old replica owns an in-flight job and exits before acknowledging it, the
new replica continues refreshing the same consumer heartbeat. The consumer
never appears expired, so Apalis cannot identify and re-enqueue the abandoned
job. For an EVM status-check job, this can leave the transaction record in
Submitted indefinitely even though the transaction has already reached a
final state on-chain.

This is consistent with the duplicate-worker-ID limitation discussed in Apalis:

• Re-enqueue task if server dies during processing. apalis-dev/apalis#504
• fix: reenqueue oprphaned before starting streaming apalis-dev/apalis#507 (comment)

In that discussion, the Apalis maintainer states that two workers with the same
name should not be run concurrently because the Redis backend cannot distinguish
their jobs and the resulting behavior is undefined.

We observed this in an ECS rolling replacement. A group of transactions was
submitted immediately around the old/new task overlap. The transactions were
confirmed on-chain, but their Redis bodies and status indexes remained
Submitted. There was no remaining status-check path that would reconcile them.

This is different from the stale-index problem fixed by #823: both the stored
transaction body and its status index genuinely remained non-final. It is also
separate from EVM nonce-counter recovery in #831.

Steps to reproduce

1. Configure the Relayer to use the Redis queue and Redis transaction
   repository.
2. Start replica A. Its EVM status worker registers as
   transaction_status_checker_evm.
3. Start replica B against the same Redis namespace. It registers with the same
   worker ID.
4. After both replicas are running, let replica A claim an EVM status-check job
   whose handler remains in flight long enough to interrupt.
5. Terminate replica A before the job is acknowledged, while replica B remains
   running and continues its heartbeat.
6. Wait longer than Apalis's configured orphan threshold.
7. Inspect the Redis consumer and in-flight keys and the transaction record.

Expected:

• Replica A has a distinct consumer identity.
• After A disappears, its in-flight job is detected as orphaned and returned to
  the queue for at-least-once processing.
• The transaction eventually reaches its final stored status.

Actual:

• Replica B refreshes the same fixed consumer identity used by A.
• The shared consumer does not expire, so A's abandoned job is not reliably
  recoverable as an orphan.
• The transaction can remain Submitted indefinitely.

Application logs

The affected task received and handled the normal ECS stop signal:

INFO SIGTERM received; starting graceful shutdown
INFO graceful worker shutdown; finishing 1 connections
INFO HTTP server stopped; draining pipeline workers
INFO Redis backend: broadcasting shutdown signal to Apalis monitors

Graceful shutdown reduces the chance of abandonment but does not make shared
worker IDs safe. The current Apalis monitor also has a five-second shutdown
timeout. Crashes, forced termination, host failure, or handlers that exceed the
drain budget still require orphan recovery to work correctly.

Platform(s)

Linux (x86)

Deployment Type

Docker container, built from source, running as overlapping replicas during an
AWS ECS rolling deployment.

Version Information

Observed on a source build whose Redis queue implementation matches current
OpenZeppelin Relayer main for the relevant worker naming and shutdown paths.
Both the observed build and current main resolve apalis/apalis-redis to
0.7.4.

Current main was also inspected and still uses the fixed worker IDs described
above.

Network Type

EVM

Suggested fix

Give every concurrently running Redis worker a unique, stable-for-process ID,
while retaining the role prefix for observability. For example:

transaction_status_checker_evm:<process-or-task-uuid>

The unique suffix should be generated once per process start and applied to all
ordinary and dynamically created Redis workers. A regression test could run two
workers against one Redis namespace, terminate the owner of an in-flight job,
and verify that the surviving worker receives the orphan after the threshold.

The watchdog proposed in #871 would be a valuable second line of defense for
lost status checks, but it does not remove the need for unique worker IDs. It
would also be useful to document the worker-ID constraint and rolling-deployment
shutdown expectations.

Code of Conduct

• I agree to follow the Code of Conduct