Proposal: Splitting CI pipelines into PR and Nightly workflows to accelerate development

[kfir4444]

Context & Problem
Currently, our Continuous Integration (CI) pipeline installs multiple external software independent of ARC, with more joining soon, and run comprehensive tests for every PR. While this ensures thorough testing, the prolonged CI times are becoming a bottleneck. Waiting for heavy environments to build and extensive test suites to finish significantly slows down the development cycle and delays crucial feedback on Pull Requests.

Proposed Solution
To expedite dev time without sacrificing code reliability, I propose we pivot to a two-tier CI strategy:

1. PR-Level CI (Fast Feedback)

• Trigger: Runs on every Pull Request and commit.
• Scope: Focuses on verifying core logic by running unit tests and mocking the heavy external software interfaces.
• Goal: Provide rapid, actionable feedback to developers (ideally in just a few minutes) to maintain momentum without getting bogged down by dependency installations.

2. Global / Nightly CI (Comprehensive Testing)

• Trigger: Runs on a scheduled basis (e.g., once a day overnight).
• Scope: Performs the full testing suite. It will build the complete environment, install all third-party software, and run the real integration tests to ensure everything interacts correctly.
• Goal: Catch deeper integration issues, edge cases, and regressions that the mocked PR tests might miss, without blocking active daily development.

Key Benefits

• Faster Iterations: Developers can merge smaller, frequent PRs without waiting on bloated pipelines.
• Resource Efficiency: We save significant compute time by avoiding redundant, heavy software installations on every minor commit.

Discussion
I would love to hear everyone’s thoughts on this pivot. A few specific questions to kick off the discussion:

• What is our preferred approach for mocking the specific software dependencies in the PR pipeline?
• Is a once-a-day schedule sufficient for the Global CI, or should it also trigger automatically on merges to main?
  @alongd @calvinp0

[calvinp0]

The idea is fantastic, I definitely am supportive of it. just the implementation will be quite extensive if we go about making fake interface shaped objects that return realistic values and excepts for external integration stuff like AutoTST/xTB/CREST etc.

I do think then if we go ahead with this, need to properly plan out pytest markers, like:

• unit
• integration
• external
• tool-specific (xtb, crest, autotst etc.)

We will also need some level of branch protection. Something along the lines of:

• build/compile arc
• min. environment install needed
• not installing external stuff and ml/ts packages
• cancel superseded run when a pr receives another commit
• maybe trigger on PR but avoid triggering push for PR branches which i gues duplicates each run?

for the nightly ci, we could split it into several parrallel jobs? example:

• Nightly: ARC + xTB/CREST
• Nightly: ARC + AutoTST
• Nightly: ARC + KinBot
• Nightly: ARC + TS-GCN
• Nightly: ARC + TorchANI/Sella
• Nightly: full ARC functional tests

That way we would get a report on what actually failed and easier to diagnose?

also, post-merge. i do think we should execute some impotats tests using real external software so we can get quick feedback if something actually broke rather than wwaiting for the nightly ci?

[calvinp0]

I think at the very least we can separate the unit tests and functional tests to run at the same time like I do with TCKDB: https://github.com/TCKDB/TCKDB/actions/runs/29942297903