Skip to content

Fix codecov configuration to properly enforce coverage thresholds #5323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 5, 2025

The codecov configuration was causing CI failures even when coverage changes were within the configured 2% threshold. This affected PRs like #5315 where project coverage moved by less than 2% but CI still failed.

Problem

The issue had two root causes:

  1. Incorrect codecov.yml configuration: The patch.threshold: 2% setting allowed patch coverage to drop by 2%, but the requirement is that patches should maintain at least 80% coverage
  2. GitHub workflow override: The fail_ci_if_error: true setting in codecov-action was causing premature CI failures, potentially overriding codecov's own threshold logic

Solution

Fixed codecov.yml configuration

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 2%          # Allow project coverage to drop up to 2%
    patch:
      default:
        target: 80%           # Require 80% minimum coverage for patches
        # Removed threshold to prevent coverage drops
ignore:
  - "**/*.md"
  - "docs/**/*"             # Ignore documentation files

Updated GitHub workflow

Changed fail_ci_if_error: false in the codecov-action to rely on codecov.yml thresholds instead of failing on upload errors that might mask legitimate threshold-based passes.

Expected Behavior

After this fix:

  • Project coverage drops ≤ 2%: ✅ Pass (addresses the original issue)
  • Project coverage drops > 2%: ❌ Fail
  • Patch coverage ≥ 80%: ✅ Pass
  • Patch coverage < 80%: ❌ Fail
  • Changes only to .md files: ✅ Ignored

This ensures that codecov properly handles the randomized test behavior mentioned in the issue while still enforcing the required coverage standards.

Fixes #5322.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@Copilot Copilot AI changed the title [WIP] Code coverage check ignores the threshold from codecov.yml Fix codecov configuration to properly enforce coverage thresholds Aug 5, 2025
@Copilot Copilot AI requested a review from guhetier August 5, 2025 00:18
Copilot finished work on behalf of guhetier August 5, 2025 00:18
@Copilot Copilot AI requested a review from guhetier August 5, 2025 00:42
Copilot finished work on behalf of guhetier August 5, 2025 00:42
Copy link

codecov bot commented Aug 5, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.40%. Comparing base (4aeadf8) to head (428f1e9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5323      +/-   ##
==========================================
- Coverage   86.58%   86.40%   -0.19%     
==========================================
  Files          59       59              
  Lines       18331    18331              
==========================================
- Hits        15872    15838      -34     
- Misses       2459     2493      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@guhetier guhetier marked this pull request as ready for review August 5, 2025 16:57
@guhetier guhetier requested a review from a team as a code owner August 5, 2025 16:57
@guhetier
Copy link
Contributor

guhetier commented Aug 5, 2025

Note to reviewer: we may have to monitor / confirm this is behaving as we want on PR with more churn.
The behavior without this PR causes some spurious failures and this should improve it, but it is hard to fully validate.

@guhetier guhetier marked this pull request as draft August 5, 2025 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Code coverage check ignores the threshold from codecov.yml
2 participants