feat(dlq): retry failed tasks from the dead-letter queue - #167
Open
GabriFedi97 wants to merge 7 commits into
Open
feat(dlq): retry failed tasks from the dead-letter queue#167GabriFedi97 wants to merge 7 commits into
GabriFedi97 wants to merge 7 commits into
Conversation
GabriFedi97
commented
Aug 24, 2026
Comment on lines
+93
to
+98
| // TODO: make the latest-uploaded-WAL marker monotonic. Retention treats | ||
| // this record as a high-water mark and won't delete tier1 WALs newer | ||
| // than it. A CLI retry re-injects an older WAL, regressing the marker; | ||
| // it fails safe (retention just gets more conservative and self-heals) | ||
| // but can briefly stall tier1 reclamation. Fix: only advance when | ||
| // t.WALName is lexicographically greater than the stored value. |
Member
Author
There was a problem hiding this comment.
This is actually not safe and needs to be fixed. Now that we allow holes in the WAL sequence, the logic of gating the deletion of WALs based on the latest uploaded WAL is not correct anymore. If WAL X and X+1 fail to upload, but X+2 is uploaded successfully, the post-backup maintenance process may delete WAL X and X+1 even if they have not been uploaded to tier2 yet. The consistency that we need to preserve here is that WAL files are not deleted by the maintenance process until they get uploaded to the tier2, if available and enabled.
Member
Author
There was a problem hiding this comment.
I believe this requires a broader change to the logic we currently have to prevent tier1 WAL deletion. As we can't rely on WAL files tier2 uploading order anymore, we may have to pin single not-yet-uploaded WAL files through another logic. Since this bug is not getting introduced by this PR, I'd suggest to open a follow-up issue and address it with an independent patch.
I have updated the TODO message: 747e204
Operators can re-enqueue WAL tasks that exhausted their delivery budget and landed in the dead-letter queue, through the new `klio admin queue wal retry [cluster-name] [wal-file...]` command: with no arguments every failed WAL is retried, a cluster name scopes the retry to that cluster, and WAL file names restrict it further. WAL names are only unique within a cluster, so passing WAL files requires a cluster; unknown WAL names are skipped rather than failing the request. Retried tasks are republished onto the work queue carrying a generic `Klio-Task-Origin: dlq-retry` header. On successful re-processing the consumer purges the matching dead-letter entries, which also clears any duplicate entries for the same WAL. The DLQ sequence stays in the JSON listing but is no longer needed to drive a retry. Assisted-by: Claude Opus 4.8 Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
Operators can re-enqueue backup tasks that landed in the dead-letter queue through the new `klio admin queue backup retry [cluster-name]` command: with no arguments every failed backup is retried, and a cluster name scopes the retry to that cluster. Multiple failed backups for the same cluster collapse into a single retry, since reprocessing one backup covers the whole cluster. Retried tasks are republished carrying the Klio-Task-Origin: dlq-retry header, mirroring the WAL retry flow. Assisted-by: Claude Opus 4.8 Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
A CLI retry can regress the per-cluster latest-uploaded-WAL marker that retention relies on as a high-water mark. Add a TODO describing the failure mode, why it fails safe, and the intended monotonic fix. Assisted-by: Claude Opus 4.8 Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
WithWALs was silently ignored by ListFailedWALTasks and RetryFailedBackupTasks instead of filtering or erroring. ListFailedWALTasks now applies the filter, and backup operations reject WithWALs since a backup task has no WAL name to filter on. StreamManager also now reuses a single cached JetStream client instead of constructing one per retried task. Assisted-by: Claude Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Signed-off-by: Gabriele Fedi <gabriele.fedi@enterprisedb.com>
GabriFedi97
marked this pull request as ready for review
August 25, 2026 07:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an operator-facing way to retry tasks that exhausted their delivery budget and landed in the dead-letter queue (DLQ), for both WAL and backup tasks:
klio admin queue wal retry [cluster-name] [WAL1 WAL2 ...]— retry every failed WAL, a single cluster's, or specific WAL files.klio admin queue backup retry [cluster-name]— retry every failed backup or a single cluster's.Retried tasks are re-published onto the work queue carrying a generic
Klio-Task-Origin: dlq-retryheader. On successful re-processing the consumer purges the matching DLQ entries (WAL: the exact DLQ entries the retry resolved; backup: thecluster's entries, since a successful backup already supersedes prior failures for that cluster).
Includes unit tests for the retry paths and regenerated CLI docs.
Design change: retries are no longer keyed on the DLQ sequence number
The original plan was to retry a failed task by passing its DLQ sequence number (as surfaced by list-failed). We moved away from that: cluster + WAL name is a more natural user-facing key, and the rest of the pipeline (dedup, purge-on-success) already identifies tasks that way. Consequences of the pivot:
Klio-Task-Origin: dlq-retryheader rather than the DLQ sequence carried in a header — it stays meaningful regardless of who triggered the retry and lets the same mechanism serve backups.wal list-failed, as it's no onger needed to drive a retry (it remains in the --json output for context/debugging).closes #166