bug: distributed run lease expired: worker ... accepted the task claim but stopped reporting to the owner coordinator (different from #2136)

[pommetjehorlepiep]

Describe the bug

Error is same as for #2136, but this is a parallel execution issue.

Workflow executes 2 sub-DAGs in parallel and both have a step that fails with distributed run lease expired: worker ... accepted the task claim but stopped reporting to the owner coordinator

sub-Dag 1 step error:

sub-Dag 2 step error:

If you look closely at the error message, you notice that both errors show the same worker!

To Reproduce

Steps to reproduce the behavior:

1. Have a k8s cluster with multiple worker nodes (maybe 1 worker node is enough?)
2. Run a workflow with multiple sub-DAGs in parallel (ideally I think with a step that runs for a minute or so)
3. See error

Expected behavior

1. sub-DAGs running in parallel are executed without issue (ie as if there is nothing else running)
2. if multiple worker nodes are available, sub-DAGs should spread over available nodes

Actual behavior

Dagu gets confused or conflicting signals by/from sub-DAGs running parallel on the same worker node
and marks steps as failed.

Environment

• Dagu version: 2.10.4
• OS: Ubuntu 24.04
• Go version: N/A
• Installation method: Dagu deployed using Helm chart. 5 node k8s cluster, with Dagu configured with 5 worker nodes

DAG configuration

n/a

Screenshots

See above

Additional context

Coordinator log

time=2026-07-11T00:15:02.132+10:00 level=INFO msg="Handler Dispatch called" run-id=033n603qqjhn09b778RiHC target=nightly-backups operation=OPERATION_RETRY
time=2026-07-11T00:15:02.328+10:00 level=INFO msg="Reusing existing queued attempt for dispatched task" run-id=033n603qqjhn09b778RiHC target=nightly-backups attempt-id=09b74b attempt-key=80d13d67032114e1:09b74b
time=2026-07-11T00:16:05.246+10:00 level=WARN msg="Marked stale distributed run as FAILED" dag=backup-3 run-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR reason="distributed run lease expired: worker dagu-worker-general-54bcbddd55-m5cbz accepted the task claim but stopped reporting to the owner coordinator"
time=2026-07-11T00:16:46.305+10:00 level=WARN msg="Marked stale distributed run as FAILED" dag=backup-2 run-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 reason="distributed run lease expired: worker dagu-worker-general-54bcbddd55-m5cbz accepted the task claim but stopped reporting to the owner coordinator"
time=2026-07-11T00:17:22.506+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-54bcbddd55-m5cbz reason="stale attempt: lease no longer active" run-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR attempt-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR attempt-key=0f437f98f6a15d78:HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR reported-status=running latest-attempt-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR latest-attempt-key=0f437f98f6a15d78:HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR latest-status=failed
time=2026-07-11T00:17:22.562+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-54bcbddd55-m5cbz reason="stale attempt: run already terminal" run-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR attempt-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR attempt-key=0f437f98f6a15d78:HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR reported-status=aborted latest-attempt-id=HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR latest-attempt-key=0f437f98f6a15d78:HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR latest-status=failed
time=2026-07-11T00:17:46.208+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-54bcbddd55-m5cbz reason="stale attempt: lease no longer active" run-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 attempt-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 attempt-key=013a4eaf9671d549:D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 reported-status=running latest-attempt-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 latest-attempt-key=013a4eaf9671d549:D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 latest-status=failed
time=2026-07-11T00:17:46.212+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-54bcbddd55-m5cbz reason="stale attempt: run already terminal" run-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 attempt-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 attempt-key=013a4eaf9671d549:D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 reported-status=aborted latest-attempt-id=D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 latest-attempt-key=013a4eaf9671d549:D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 latest-status=failed

Worker log

time=2026-07-11T00:15:02.667+10:00 level=INFO msg="Task polled successfully" run-id=033n603qqjhn09b778RiHC target=nightly-backups worker-selector=map[] coordinator-id=coord-0
time=2026-07-11T00:15:02.667+10:00 level=INFO msg="Task received" worker-id=dagu-worker-general-54bcbddd55-m5cbz poller-id=331884a3-3bf2-47f6-b594-e6721e2889dc poller-index=99 root-dag-run-name=nightly-backups root-dag-run-id=033n603qqjhn09b778RiHC parent-dag-run-name="" parent-dag-run-id="" run-id=033n603qqjhn09b778RiHC
time=2026-07-11T00:15:02.667+10:00 level=INFO msg="Task received, starting execution" worker-id=dagu-worker-general-54bcbddd55-m5cbz poller-index=99 run-id=033n603qqjhn09b778RiHC
time=2026-07-11T00:15:02.893+10:00 level=INFO msg="Executing remote task" operation=OPERATION_RETRY target=nightly-backups run-id=033n603qqjhn09b778RiHC root-dag-run-id=033n603qqjhn09b778RiHC parent-dag-run-id=""
time=2026-07-11T00:15:02.894+10:00 level=INFO msg="Using previous status from task for retry" run-id=033n603qqjhn09b778RiHC nodes=6
time=2026-07-11T00:15:02.894+10:00 level=INFO msg="Creating temporary DAG file from definition" dag=nightly-backups size=641
{"time":"2026-07-11T00:15:03.545022752+10:00","level":"INFO","msg":"DAG run retry started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","retry-target-attempt-id":"09b74b"}
{"time":"2026-07-11T00:15:03.545336956+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1"}
{"time":"2026-07-11T00:15:03.545390067+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2"}
{"time":"2026-07-11T00:15:03.545416905+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_3"}
{"time":"2026-07-11T00:15:03.545441863+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4"}
{"time":"2026-07-11T00:15:03.546109603+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","dag":"backup-4"}
{"time":"2026-07-11T00:15:03.555675874+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","dag":"backup-4"}
{"time":"2026-07-11T00:15:03.812665948+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","dag":"backup-1"}
{"time":"2026-07-11T00:15:03.821854961+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","dag":"backup-1"}
{"time":"2026-07-11T00:15:04.466249589+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","dag":"backup-2"}
{"time":"2026-07-11T00:15:04.481346268+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","dag":"backup-2"}
{"time":"2026-07-11T00:15:04.732789067+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_3","step":"backup_3","dag":"backup-3"}
{"time":"2026-07-11T00:15:04.744531949+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_3","step":"backup_3","dag":"backup-3"}
{"time":"2026-07-11T00:15:04.989037332+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq"}
{"time":"2026-07-11T00:15:05.000589663+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","file":"/data/logs/3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq.log","err":"open /data/logs/3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq.log: no such file or directory"}
{"time":"2026-07-11T00:15:05.233732316+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM"}
{"time":"2026-07-11T00:15:05.243578167+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","file":"/data/logs/8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM.log","err":"open /data/logs/8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM.log: no such file or directory"}
{"time":"2026-07-11T00:15:05.686174862+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7"}
{"time":"2026-07-11T00:15:05.695652551+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","file":"/data/logs/D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7.log","err":"open /data/logs/D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7.log: no such file or directory"}
{"time":"2026-07-11T00:15:05.823370942+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","params":[]}
{"time":"2026-07-11T00:15:05.823698808+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"execute_backup_4"}
{"time":"2026-07-11T00:15:05.823750659+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_script_run_btbw_job"}
{"time":"2026-07-11T00:15:05.823822461+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_k8s_backup_job"}
{"time":"2026-07-11T00:15:05.994110323+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_3","step":"backup_3","sub-dag":"backup-3","sub-run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR"}
{"time":"2026-07-11T00:15:06.394017132+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","params":[]}
{"time":"2026-07-11T00:15:06.394286573+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"scale_down"}
{"time":"2026-07-11T00:15:06.394327761+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_script_run_btbw_job"}
{"time":"2026-07-11T00:15:06.394352989+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_k8s_backup_job"}
{"time":"2026-07-11T00:15:06.89708887+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","params":[]}
{"time":"2026-07-11T00:15:06.897671136+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"scale_down"}
{"time":"2026-07-11T00:15:06.897880162+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_script_run_btbw_job"}
{"time":"2026-07-11T00:15:06.898012806+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_k8s_backup_job"}
{"time":"2026-07-11T00:15:07.594512493+10:00","level":"INFO","msg":"DAG run started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","params":[]}
{"time":"2026-07-11T00:15:07.59483387+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"get_day_of_week"}
{"time":"2026-07-11T00:15:07.594883011+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_script_run_btbw_job"}
{"time":"2026-07-11T00:15:07.594909141+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"download_k8s_backup_job"}
{"time":"2026-07-11T00:15:09.194879585+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"full_or_incremental_backup"}
{"time":"2026-07-11T00:15:09.798480504+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"full_backup"}
{"time":"2026-07-11T00:15:09.798568114+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"incremental_backup"}
{"time":"2026-07-11T00:15:10.294745694+10:00","level":"INFO","msg":"Checking preconditions for \"full_backup\"","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"full_backup","step":"full_backup"}
{"time":"2026-07-11T00:15:10.493788284+10:00","level":"INFO","msg":"Preconditions failed for \"full_backup\"","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"full_backup","step":"full_backup"}
{"time":"2026-07-11T00:15:10.694941465+10:00","level":"INFO","msg":"Checking preconditions for \"incremental_backup\"","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"incremental_backup","step":"incremental_backup"}
{"time":"2026-07-11T00:15:11.89668543+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"set_script_permissions"}
{"time":"2026-07-11T00:15:12.093676245+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"set_script_permissions"}
{"time":"2026-07-11T00:15:12.095252914+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"set_script_permissions"}
{"time":"2026-07-11T00:15:12.894756407+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-11T00:15:13.198652332+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"wait_till_scaled_down"}
{"time":"2026-07-11T00:15:13.496590914+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"set_script_permissions"}
{"time":"2026-07-11T00:15:14.084414466+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"wait_till_scaled_down"}
{"time":"2026-07-11T00:15:43.983536866+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","name":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","attempt-id":"","dag":"backup-4","run-id":"3T6xGt39U1C1nghKFUju9cURSRqX73G6TPfXqyP654mq","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","status":"succeeded","started-at":"2026-07-11T00:15:05+10:00","finished-at":"2026-07-11T00:15:43+10:00"}
{"time":"2026-07-11T00:15:44.082838284+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-11T00:15:44.474783797+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-11T00:16:18.453348653+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"scale_up"}
{"time":"2026-07-11T00:16:21.07577749+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","name":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","attempt-id":"","dag":"backup-1","run-id":"8gPvxhGxM9HF6FFSuV2UMHYBztLcrU4BwrdnZUE1nSMM","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","status":"succeeded","started-at":"2026-07-11T00:15:05+10:00","finished-at":"2026-07-11T00:16:21+10:00"}
{"time":"2026-07-11T00:17:22.491539717+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-11T00:17:22.505049508+10:00","level":"ERROR","msg":"Failed to push status to coordinator","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","err":"status rejected: stale attempt: lease no longer active"}
{"time":"2026-07-11T00:17:22.505147337+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","attempt-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","reason":"stale attempt: lease no longer active"}
time=2026-07-11T00:17:22.505+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-11T00:17:22.505+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-11T00:17:22.505555086+10:00","level":"INFO","msg":"DAG run finished","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","status":"aborted","started-at":"2026-07-11T00:15:06+10:00","finished-at":"2026-07-11T00:17:22+10:00"}
{"time":"2026-07-11T00:17:22.561630981+10:00","level":"ERROR","msg":"Failed to push status to coordinator","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","err":"status rejected: stale attempt: run already terminal"}
{"time":"2026-07-11T00:17:22.561656291+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","dag":"backup-3","run-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","attempt-id":"HG7wubrDy2UYeVLkpg7t4F9NhugFbMe5NLEh2yYmVSFR","reason":"stale attempt: run already terminal"}
time=2026-07-11T00:17:22.561+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-11T00:17:22.561+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-11T00:17:46.200852881+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","step":"scale_up"}
{"time":"2026-07-11T00:17:46.206892816+10:00","level":"ERROR","msg":"Failed to push status to coordinator","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","err":"status rejected: stale attempt: lease no longer active"}
{"time":"2026-07-11T00:17:46.206977835+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","attempt-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","reason":"stale attempt: lease no longer active"}
time=2026-07-11T00:17:46.207+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-11T00:17:46.207+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-11T00:17:46.207423826+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","status":"aborted","started-at":"2026-07-11T00:15:05+10:00","finished-at":"2026-07-11T00:17:46+10:00"}
{"time":"2026-07-11T00:17:46.211060676+10:00","level":"ERROR","msg":"Failed to push status to coordinator","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","err":"status rejected: stale attempt: run already terminal"}
{"time":"2026-07-11T00:17:46.211138291+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","name":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","attempt-id":"","dag":"backup-2","run-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","root":"nightly-backups:033n603qqjhn09b778RiHC","parent":"nightly-backups:033n603qqjhn09b778RiHC","attempt-id":"D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7","reason":"stale attempt: run already terminal"}
time=2026-07-11T00:17:46.211+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-11T00:17:46.211+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-11T00:17:46.267343666+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","attempt-id":"09b74b","dag":"nightly-backups","run-id":"033n603qqjhn09b778RiHC","status":"failed","started-at":"2026-07-11T00:15:02+10:00","finished-at":"2026-07-11T00:17:46+10:00"}
{"time":"2026-07-11T00:17:46.689639719+10:00","level":"ERROR","msg":"DAG execution failed","run-id":"033n603qqjhn09b778RiHC","err":"sub DAG run D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 failed with status: aborted"}
time=2026-07-11T00:17:46.690+10:00 level=ERROR msg="Task execution failed" worker-id=dagu-worker-general-54bcbddd55-m5cbz poller-index=99 run-id=033n603qqjhn09b778RiHC err="sub DAG run D8PFQxZVvwNb31kZfbPGh7UZqjuKeF1boppUb9qfUWg7 failed with status: aborted"

[pommetjehorlepiep]

@yohamta0, thanks for the quick fixes.

I have retested with Dagu 2.10.6, and unfortunately same issue still existing.

coordinator

time=2026-07-12T21:00:05.122+10:00 level=INFO msg="Coordinator initialization" bind-address=0.0.0.0 advertise-address=dagu-coordinator.default.svc.k8s.cluster port=50055 instance-id=dagu-coordinator-5dc67ffb84-jljhp@50055
time=2026-07-12T21:00:05.122+10:00 level=INFO msg="Started zombie detector" interval=45s
time=2026-07-12T21:00:05.122+10:00 level=INFO msg="Starting service registry" service=coordinator service-id=dagu-coordinator-5dc67ffb84-jljhp@50055 host=dagu-coordinator.default.svc.k8s.cluster port=50055 status=active
time=2026-07-12T21:00:05.122+10:00 level=INFO msg="Starting health check server" service=coordinator port=8091
time=2026-07-12T21:00:05.313+10:00 level=INFO msg="Registered with service registry" service-id=dagu-coordinator-5dc67ffb84-jljhp@50055 configured-host=dagu-coordinator.default.svc.k8s.cluster port=50055 addr=[::]:50055
time=2026-07-12T21:00:05.313+10:00 level=INFO msg="Starting to serve on coordinator service" addr=[::]:50055
time=2026-07-12T22:00:03.198+10:00 level=INFO msg="Handler Dispatch called" run-id=033oBMh11Ic8OUX6QsGarS target=freshrss-update-feeds operation=OPERATION_RETRY
time=2026-07-12T22:00:03.256+10:00 level=INFO msg="Reusing existing queued attempt for dispatched task" run-id=033oBMh11Ic8OUX6QsGarS target=freshrss-update-feeds attempt-id=89eb5d attempt-key=b0a0dca7b16ec2ed:89eb5d
time=2026-07-12T23:00:02.973+10:00 level=INFO msg="Handler Dispatch called" run-id=033oCpzBLzJCUGE6vW612j target=freshrss-update-feeds operation=OPERATION_RETRY
time=2026-07-12T23:00:03.062+10:00 level=INFO msg="Reusing existing queued attempt for dispatched task" run-id=033oCpzBLzJCUGE6vW612j target=freshrss-update-feeds attempt-id=97b150 attempt-key=fd5fc8bbb3e05f87:97b150
time=2026-07-12T23:30:02.945+10:00 level=INFO msg="Handler Dispatch called" run-id=033oDZdElwjVF4ACFfHYdq target=clamav-update-virus-database operation=OPERATION_RETRY
time=2026-07-12T23:30:03.038+10:00 level=INFO msg="Reusing existing queued attempt for dispatched task" run-id=033oDZdElwjVF4ACFfHYdq target=clamav-update-virus-database attempt-id=3cbd99 attempt-key=de62c5efd0f9649b:3cbd99
time=2026-07-13T00:15:03.045+10:00 level=INFO msg="Handler Dispatch called" run-id=033oEg6MKFuUvZf0ojawVl target=nightly-backups operation=OPERATION_RETRY
time=2026-07-13T00:15:03.187+10:00 level=INFO msg="Reusing existing queued attempt for dispatched task" run-id=033oEg6MKFuUvZf0ojawVl target=nightly-backups attempt-id=1f1551 attempt-key=be3ea985faed074d:1f1551
time=2026-07-13T00:15:52.181+10:00 level=WARN msg="Marked stale distributed run as FAILED" dag=backup-1 run-id=FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ reason="distributed run lease expired: worker dagu-worker-general-5db6d47565-pzlzs accepted the task claim but stopped reporting to the owner coordinator"
time=2026-07-13T00:15:55.427+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-5db6d47565-pzlzs reason="stale attempt: run already terminal" run-id=FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ attempt-id=FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ attempt-key=a13508bba6148918:FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ reported-status=succeeded latest-attempt-id=FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ latest-attempt-key=a13508bba6148918:FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ latest-status=failed
time=2026-07-13T00:16:37.032+10:00 level=WARN msg="Marked stale distributed run as FAILED" dag=backup-2 run-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn reason="distributed run lease expired: worker dagu-worker-general-5db6d47565-pzlzs accepted the task claim but stopped reporting to the owner coordinator"
time=2026-07-13T00:16:37.845+10:00 level=WARN msg="Marked stale distributed run as FAILED" dag=backup-3 run-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ reason="distributed run lease expired: worker dagu-worker-general-5db6d47565-pzlzs accepted the task claim but stopped reporting to the owner coordinator"
time=2026-07-13T00:17:43.593+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-5db6d47565-pzlzs reason="stale attempt: lease no longer active" run-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn attempt-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn attempt-key=21b23c3e49da3b2a:9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn reported-status=running latest-attempt-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn latest-attempt-key=21b23c3e49da3b2a:9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn latest-status=failed
time=2026-07-13T00:17:43.598+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-5db6d47565-pzlzs reason="stale attempt: run already terminal" run-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn attempt-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn attempt-key=21b23c3e49da3b2a:9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn reported-status=aborted latest-attempt-id=9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn latest-attempt-key=21b23c3e49da3b2a:9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn latest-status=failed
time=2026-07-13T00:18:12.504+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-5db6d47565-pzlzs reason="stale attempt: lease no longer active" run-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ attempt-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ attempt-key=71f6f5d03bdb9ad3:G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ reported-status=running latest-attempt-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ latest-attempt-key=71f6f5d03bdb9ad3:G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ latest-status=failed
time=2026-07-13T00:18:12.638+10:00 level=WARN msg="Rejected remote status update" worker-id=dagu-worker-general-5db6d47565-pzlzs reason="stale attempt: run already terminal" run-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ attempt-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ attempt-key=71f6f5d03bdb9ad3:G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ reported-status=aborted latest-attempt-id=G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ latest-attempt-key=71f6f5d03bdb9ad3:G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ latest-status=failed

worker (dagu-worker-general-5db6d47565-pzlzs)

time=2026-07-13T00:15:03.988+10:00 level=INFO msg="Task polled successfully" run-id=033oEg6MKFuUvZf0ojawVl target=nightly-backups worker-selector=map[] coordinator-id=coord-0
time=2026-07-13T00:15:03.988+10:00 level=INFO msg="Task received" worker-id=dagu-worker-general-5db6d47565-pzlzs poller-id=1053cd0b-c7b0-414f-a764-e52fd32d0492 poller-index=70 root-dag-run-name=nightly-backups root-dag-run-id=033oEg6MKFuUvZf0ojawVl parent-dag-run-name="" parent-dag-run-id="" run-id=033oEg6MKFuUvZf0ojawVl
time=2026-07-13T00:15:03.988+10:00 level=INFO msg="Task received, starting execution" worker-id=dagu-worker-general-5db6d47565-pzlzs poller-index=70 run-id=033oEg6MKFuUvZf0ojawVl
time=2026-07-13T00:15:04.394+10:00 level=INFO msg="Executing remote task" operation=OPERATION_RETRY target=nightly-backups run-id=033oEg6MKFuUvZf0ojawVl root-dag-run-id=033oEg6MKFuUvZf0ojawVl parent-dag-run-id=""
time=2026-07-13T00:15:04.395+10:00 level=INFO msg="Using previous status from task for retry" run-id=033oEg6MKFuUvZf0ojawVl nodes=6
time=2026-07-13T00:15:04.395+10:00 level=INFO msg="Creating temporary DAG file from definition" dag=nightly-backups size=641
{"time":"2026-07-13T00:15:05.519190935+10:00","level":"INFO","msg":"DAG run retry started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","retry-target-attempt-id":"1f1551"}
{"time":"2026-07-13T00:15:05.519504529+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4"}
{"time":"2026-07-13T00:15:05.519555923+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2"}
{"time":"2026-07-13T00:15:05.519583196+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_3"}
{"time":"2026-07-13T00:15:05.519607515+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1"}
{"time":"2026-07-13T00:15:05.520299156+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","dag":"backup-1"}
{"time":"2026-07-13T00:15:05.528981553+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","dag":"backup-1"}
{"time":"2026-07-13T00:15:06.31763619+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","dag":"backup-4"}
{"time":"2026-07-13T00:15:06.327632557+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","dag":"backup-4"}
{"time":"2026-07-13T00:15:07.458171293+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","dag":"backup-2"}
{"time":"2026-07-13T00:15:07.468594232+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","dag":"backup-2"}
{"time":"2026-07-13T00:15:07.990027285+10:00","level":"INFO","msg":"No local DAG store, trying remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_3","step":"backup_3","dag":"backup-3"}
{"time":"2026-07-13T00:15:08.000734742+10:00","level":"INFO","msg":"DAG loaded from remote fallback","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_3","step":"backup_3","dag":"backup-3"}
{"time":"2026-07-13T00:15:08.546373223+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ"}
{"time":"2026-07-13T00:15:08.557725666+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","file":"/data/logs/FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ.log","err":"open /data/logs/FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ.log: no such file or directory"}
{"time":"2026-07-13T00:15:09.030014101+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx"}
{"time":"2026-07-13T00:15:09.125071539+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","file":"/data/logs/FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx.log","err":"open /data/logs/FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx.log: no such file or directory"}
{"time":"2026-07-13T00:15:09.965240497+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn"}
{"time":"2026-07-13T00:15:09.975513131+10:00","level":"WARN","msg":"Failed to open sub DAG scheduler log file","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","file":"/data/logs/9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn.log","err":"open /data/logs/9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn.log: no such file or directory"}
{"time":"2026-07-13T00:15:10.302547775+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","params":[]}
{"time":"2026-07-13T00:15:10.302946609+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"execute_backup_1"}
{"time":"2026-07-13T00:15:10.30300892+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_script_run_btbw_job"}
{"time":"2026-07-13T00:15:10.30306118+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_k8s_backup_job"}
{"time":"2026-07-13T00:15:10.722583094+10:00","level":"INFO","msg":"Executing sub DAG via injected sub-workflow runner","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_3","step":"backup_3","sub-dag":"backup-3","sub-run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ"}
{"time":"2026-07-13T00:15:11.373146122+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","params":[]}
{"time":"2026-07-13T00:15:11.373409353+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"scale_down"}
{"time":"2026-07-13T00:15:11.373463127+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_script_run_btbw_job"}
{"time":"2026-07-13T00:15:11.37349139+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_k8s_backup_job"}
{"time":"2026-07-13T00:15:12.073350061+10:00","level":"INFO","msg":"DAG run started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","params":[]}
{"time":"2026-07-13T00:15:12.073636788+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"scale_down"}
{"time":"2026-07-13T00:15:12.073695256+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_script_run_btbw_job"}
{"time":"2026-07-13T00:15:12.073723121+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_k8s_backup_job"}
{"time":"2026-07-13T00:15:12.615461088+10:00","level":"INFO","msg":"DAG run started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","params":[]}
{"time":"2026-07-13T00:15:12.615759226+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"get_day_of_week"}
{"time":"2026-07-13T00:15:12.615855776+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_script_run_btbw_job"}
{"time":"2026-07-13T00:15:12.615890491+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"download_k8s_backup_job"}
{"time":"2026-07-13T00:15:15.715732541+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"full_or_incremental_backup"}
{"time":"2026-07-13T00:15:15.817287433+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"set_script_permissions"}
{"time":"2026-07-13T00:15:16.402582769+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"set_script_permissions"}
{"time":"2026-07-13T00:15:17.015373262+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"set_script_permissions"}
{"time":"2026-07-13T00:15:17.100703902+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"full_backup"}
{"time":"2026-07-13T00:15:17.100768411+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"incremental_backup"}
{"time":"2026-07-13T00:15:17.816242835+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-13T00:15:17.898363288+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"set_script_permissions"}
{"time":"2026-07-13T00:15:18.400080854+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"wait_till_scaled_down"}
{"time":"2026-07-13T00:15:18.684699561+10:00","level":"INFO","msg":"Checking preconditions for \"incremental_backup\"","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"incremental_backup","step":"incremental_backup"}
{"time":"2026-07-13T00:15:19.498775712+10:00","level":"INFO","msg":"Checking preconditions for \"full_backup\"","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"full_backup","step":"full_backup"}
{"time":"2026-07-13T00:15:19.51601028+10:00","level":"INFO","msg":"Preconditions failed for \"full_backup\"","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"full_backup","step":"full_backup"}
{"time":"2026-07-13T00:15:19.615613802+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"wait_till_scaled_down"}
{"time":"2026-07-13T00:15:48.85642732+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-13T00:15:49.185939407+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-13T00:15:55.418978069+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","status":"succeeded","started-at":"2026-07-13T00:15:08+10:00","finished-at":"2026-07-13T00:15:55+10:00"}
{"time":"2026-07-13T00:15:55.425370965+10:00","level":"ERROR","msg":"Failed to push status to coordinator","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","err":"status rejected: stale attempt: run already terminal"}
{"time":"2026-07-13T00:15:55.425423124+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_1","step":"backup_1","sub-dag":"backup-1","sub-run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","name":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","attempt-id":"","dag":"backup-1","run-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","attempt-id":"FqS6fsTDtZy8pwceHRDFtxC5AbDcwjpduu9TvcRXG5aJ","reason":"stale attempt: run already terminal"}
time=2026-07-13T00:15:55.425+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:15:55.425+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:16:23.706329908+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"scale_up"}
{"time":"2026-07-13T00:16:25.714475302+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_4","step":"backup_4","sub-dag":"backup-4","sub-run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","name":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","attempt-id":"","dag":"backup-4","run-id":"FYLJZmWqQfsacEE7ZA5EXLRnk7gFD1fAyvNFZtFiEKEx","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","status":"succeeded","started-at":"2026-07-13T00:15:09+10:00","finished-at":"2026-07-13T00:16:25+10:00"}
{"time":"2026-07-13T00:17:43.585892609+10:00","level":"INFO","msg":"Step started","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"scale_up"}
{"time":"2026-07-13T00:17:43.593768112+10:00","level":"ERROR","msg":"Failed to push status to coordinator","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","err":"status rejected: stale attempt: lease no longer active"}
{"time":"2026-07-13T00:17:43.593905423+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","attempt-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","reason":"stale attempt: lease no longer active"}
time=2026-07-13T00:17:43.593+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:17:43.594+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:17:43.594346032+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","status":"aborted","started-at":"2026-07-13T00:15:09+10:00","finished-at":"2026-07-13T00:17:43+10:00"}
{"time":"2026-07-13T00:17:43.597937692+10:00","level":"ERROR","msg":"Failed to push status to coordinator","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","err":"status rejected: stale attempt: run already terminal"}
{"time":"2026-07-13T00:17:43.598019979+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","step":"backup_2","step":"backup_2","sub-dag":"backup-2","sub-run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","name":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","attempt-id":"","dag":"backup-2","run-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","attempt-id":"9dViEC8XPwm3cWcgnK7QvwZFV36HA8YtDdMdrTkEhSfn","reason":"stale attempt: run already terminal"}
time=2026-07-13T00:17:43.598+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:17:43.598+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:18:12.496446537+10:00","level":"INFO","msg":"Step started","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","step":"run_backup_to_borgwarehouse_job"}
{"time":"2026-07-13T00:18:12.504524033+10:00","level":"ERROR","msg":"Failed to push status to coordinator","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","err":"status rejected: stale attempt: lease no longer active"}
{"time":"2026-07-13T00:18:12.504602002+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","attempt-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","reason":"stale attempt: lease no longer active"}
time=2026-07-13T00:18:12.504+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:18:12.504+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:18:12.505042536+10:00","level":"INFO","msg":"DAG run finished","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","status":"aborted","started-at":"2026-07-13T00:15:10+10:00","finished-at":"2026-07-13T00:18:12+10:00"}
{"time":"2026-07-13T00:18:12.638352178+10:00","level":"ERROR","msg":"Failed to push status to coordinator","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","err":"status rejected: stale attempt: run already terminal"}
{"time":"2026-07-13T00:18:12.638408376+10:00","level":"WARN","msg":"Coordinator rejected the worker attempt; stopping execution","dag":"backup-3","run-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","root":"nightly-backups:033oEg6MKFuUvZf0ojawVl","parent":"nightly-backups:033oEg6MKFuUvZf0ojawVl","attempt-id":"G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ","reason":"stale attempt: run already terminal"}
time=2026-07-13T00:18:12.638+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:18:12.638+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:18:12.708910762+10:00","level":"INFO","msg":"DAG run finished","name":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","attempt-id":"1f1551","dag":"nightly-backups","run-id":"033oEg6MKFuUvZf0ojawVl","status":"failed","started-at":"2026-07-13T00:15:04+10:00","finished-at":"2026-07-13T00:18:12+10:00"}
time=2026-07-13T00:18:12.919+10:00 level=INFO msg="Cancelling task per coordinator directive" worker-id=dagu-worker-general-5db6d47565-pzlzs attempt-key=be3ea985faed074d:1f1551
time=2026-07-13T00:18:12.919+10:00 level=INFO msg="Stopping running child processes" stop-mode=graceful signal=terminated allow-override=true max-cleanup-time=5s
time=2026-07-13T00:18:12.919+10:00 level=INFO msg="All child processes have been terminated"
{"time":"2026-07-13T00:18:13.262673747+10:00","level":"ERROR","msg":"DAG execution failed","run-id":"033oEg6MKFuUvZf0ojawVl","err":"sub DAG run G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ failed with status: aborted"}
time=2026-07-13T00:18:13.263+10:00 level=ERROR msg="Task execution failed" worker-id=dagu-worker-general-5db6d47565-pzlzs poller-index=70 run-id=033oEg6MKFuUvZf0ojawVl err="sub DAG run G7yLfmX8278AnPZSyBRYNB8QfZ9jcc2iGP8ZtRA6yNeQ failed with status: aborted"