Skip to content

fix(gitops-update): add retry with rebase + exponential backoff on push #197

@bedatty

Description

@bedatty

Context

The gitops-update.yml workflow is susceptible to race conditions when multiple services push tag updates to the same GitOps repository concurrently. When two or more CI pipelines (e.g., product-console and midaz-ledger) try to update midaz-firmino-gitops at the same time, one of them will fail with:

! [rejected]  main -> main (fetch first)
error: failed to push some refs to 'https://github.com/LerianStudio/midaz-firmino-gitops'
hint: Updates were rejected because the remote contains work that you do not have locally.

Recent failure example: https://github.com/LerianStudio/product-console/actions/runs/23855041026/job/69560895833

Root cause

The "Commit & push (GitOps)" step does a single git push origin main with no retry mechanism. Between git pull and git push, another workflow may have already pushed a commit, causing the push to be rejected.

Proposed solution

Replace the single push attempt with a retry loop using git pull --rebase and exponential backoff:

MAX_RETRIES=5
for i in $(seq 1 $MAX_RETRIES); do
  git pull --rebase origin main
  git push origin main && break
  if [ "$i" -eq "$MAX_RETRIES" ]; then
    echo "ERROR: Failed to push after $MAX_RETRIES attempts"
    exit 1
  fi
  echo "Push failed (attempt $i/$MAX_RETRIES), retrying in $((i * 2))s..."
  sleep $((i * 2))
done

Why this works

  • Tag updates in values.yaml touch different lines per service, so rebases will almost never conflict
  • Exponential backoff (2s, 4s, 6s, 8s, 10s) reduces contention
  • 5 retries gives a ~30s total window which is more than enough for concurrent pipelines to serialize
  • This is the industry-standard pattern for GitOps multi-service repos

Affected workflow

  • .github/workflows/gitops-update.yml — "Commit & push (GitOps)" step (currently at v1.17.0)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions