Skip to content

fix(gitops-update): add retry with rebase and exponential backoff on push#199

Merged
bedatty merged 1 commit intodevelopfrom
fix/gitops-push-retry-with-rebase
Apr 1, 2026
Merged

fix(gitops-update): add retry with rebase and exponential backoff on push#199
bedatty merged 1 commit intodevelopfrom
fix/gitops-push-retry-with-rebase

Conversation

@bedatty
Copy link
Copy Markdown
Contributor

@bedatty bedatty commented Apr 1, 2026

Lerian

GitHub Actions Shared Workflows


Description

The gitops-update.yml workflow currently does a single git push origin main when updating the GitOps repo. When multiple services (e.g., product-console and midaz-ledger) trigger tag updates concurrently, one push gets rejected because the remote already advanced:

! [rejected]  main -> main (fetch first)
error: failed to push some refs to '...'

This PR replaces the single push with a retry loop (up to 5 attempts) using git pull --rebase and exponential backoff (2s, 4s, 6s, 8s, 10s). Tag updates touch different lines per service, so rebases will almost never conflict.

Recent failure: https://github.com/LerianStudio/product-console/actions/runs/23855041026/job/69560895833

Type of Change

  • fix: Bug fix in a workflow (incorrect behavior, broken step, wrong condition)

Breaking Changes

None.

Testing

  • YAML syntax validated locally
  • Triggered a real workflow run on a caller repository using @develop or the beta tag
  • Verified all existing inputs still work with default values
  • Confirmed no secrets or tokens are printed in logs
  • Checked that unrelated workflows are not affected

Caller repo / workflow run: To be validated after merge to develop.

Related Issues

Closes #197

Summary by CodeRabbit

  • Chores
    • Enhanced reliability of automated deployments with retry logic for Git push operations, including exponential backoff and automatic rebase handling for concurrent upstream changes.

…push

When multiple services update the same GitOps repo concurrently, the
push can be rejected because the remote already has new commits. This
adds a retry loop (up to 5 attempts) with git pull --rebase and
exponential backoff (2s, 4s, 6s, 8s, 10s) to handle race conditions.

Closes #197
@bedatty bedatty requested a review from a team as a code owner April 1, 2026 16:47
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 1, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9c506ae2-f2cc-447e-b7b2-72c89732ad18

📥 Commits

Reviewing files that changed from the base of the PR and between f2e73da and bb34537.

📒 Files selected for processing (1)
  • .github/workflows/gitops-update.yml

Walkthrough

Updated the GitOps workflow to implement retry logic with exponential backoff on git push failures. The single push command was replaced with a loop attempting up to 5 times, incorporating git pull --rebase between attempts and pausing with exponential backoff to reduce contention during concurrent operations.

Changes

Cohort / File(s) Summary
GitOps Workflow Retry Logic
.github/workflows/gitops-update.yml
Added 5-attempt retry loop with exponential backoff (2–10 second intervals) and git pull --rebase to resolve concurrent push conflicts. Exits non-zero on final failure; logs successful attempt number.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding retry logic with rebase and exponential backoff to the GitOps push operation.
Description check ✅ Passed The description covers all required sections: rationale for the fix, type of change marked, testing performed, and related issue referenced. The concrete failure example strengthens the justification.
Linked Issues check ✅ Passed The implementation directly addresses issue #197: replaces single push with retry loop, adds git pull --rebase before each push, implements exponential backoff (2s, 4s, 6s, 8s, 10s), and caps at 5 attempts as specified.
Out of Scope Changes check ✅ Passed All changes are confined to the gitops-update.yml workflow file and directly address the concurrent push race condition. No unrelated modifications or refactoring were introduced.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gitops-push-retry-with-rebase

Comment @coderabbitai help to get the list of available commands and usage tips.

@lerian-studio lerian-studio added size/XS PR changes < 50 lines workflow Changes to one or more reusable workflow files labels Apr 1, 2026
@lerian-studio
Copy link
Copy Markdown

🔍 Lint Analysis

Check Files Scanned Status
YAML Lint 1 file(s) ✅ success
Action Lint 1 file(s) ✅ success
Pinned Actions 1 file(s) ✅ success
Markdown Link Check no changes ⏭️ skipped
Spelling Check 1 file(s) ✅ success
Shell Check 1 file(s) ✅ success
README Check 1 file(s) ✅ success
Composite Schema no changes ⏭️ skipped

🔍 View full scan logs

@lerian-studio
Copy link
Copy Markdown

🛡️ CodeQL Analysis Results

Languages analyzed: actions

Found 1 issue(s): 1 Medium

Severity Rule File Message
🟡 Medium actions/untrusted-checkout/medium .github/workflows/gitops-update.yml:83 Potential unsafe checkout of untrusted pull request on privileged workflow.

🔍 View full scan logs | 🛡️ Security tab

@bedatty bedatty merged commit 21295c9 into develop Apr 1, 2026
17 checks passed
@github-actions github-actions bot deleted the fix/gitops-push-retry-with-rebase branch April 1, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS PR changes < 50 lines workflow Changes to one or more reusable workflow files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants