-
Notifications
You must be signed in to change notification settings - Fork 0
fix(gitops-update): add retry with rebase + exponential backoff on push #197
Copy link
Copy link
Open
Description
Context
The gitops-update.yml workflow is susceptible to race conditions when multiple services push tag updates to the same GitOps repository concurrently. When two or more CI pipelines (e.g., product-console and midaz-ledger) try to update midaz-firmino-gitops at the same time, one of them will fail with:
! [rejected] main -> main (fetch first)
error: failed to push some refs to 'https://github.com/LerianStudio/midaz-firmino-gitops'
hint: Updates were rejected because the remote contains work that you do not have locally.
Recent failure example: https://github.com/LerianStudio/product-console/actions/runs/23855041026/job/69560895833
Root cause
The "Commit & push (GitOps)" step does a single git push origin main with no retry mechanism. Between git pull and git push, another workflow may have already pushed a commit, causing the push to be rejected.
Proposed solution
Replace the single push attempt with a retry loop using git pull --rebase and exponential backoff:
MAX_RETRIES=5
for i in $(seq 1 $MAX_RETRIES); do
git pull --rebase origin main
git push origin main && break
if [ "$i" -eq "$MAX_RETRIES" ]; then
echo "ERROR: Failed to push after $MAX_RETRIES attempts"
exit 1
fi
echo "Push failed (attempt $i/$MAX_RETRIES), retrying in $((i * 2))s..."
sleep $((i * 2))
doneWhy this works
- Tag updates in
values.yamltouch different lines per service, so rebases will almost never conflict - Exponential backoff (2s, 4s, 6s, 8s, 10s) reduces contention
- 5 retries gives a ~30s total window which is more than enough for concurrent pipelines to serialize
- This is the industry-standard pattern for GitOps multi-service repos
Affected workflow
.github/workflows/gitops-update.yml— "Commit & push (GitOps)" step (currently atv1.17.0)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels