Implement "dispatch" pattern from telemetry shared-actions

[msarahan]

One thing useful that came out of the telemetry work was the "dispatch" pattern, which is described in the shared-actions repo readme. The general idea is that instead of calling an action directly with a repo/branch reference, you should first manually clone the shared-actions repo at that branch, and then call the desired action with a relative path. The advantages to doing this are:

• It's much easier to include code outside of actions, such as native Python files. When a shared action is called directly, any other files in the repo do not come along for the ride. Using native files facilitates use of linters and other code helper tools.
• It's much easier to change branch references for several actions at once. In the shared-actions workflow, we run a telemetry-stash-base-env-vars action that captures several environment variables, two of which are the SHARED_ACTIONS_REPO and SHARED_ACTIONS_REF variables. This are written to a file, and stored as a github artifact. Subsequent actions (which use very stable logic on the main branch) load that file, read those env vars, clone the requested repo/ref, and call the requested shared action with relative path.
• This is a hack around the restricted list of actions that rapids jobs can clone, allowing us to point to our user forks instead of pushing branches directly to the rapidsai org repo.

For example, instead of changing multiple branch references to test a shared-workflows branch, like:

One could have:

# This is a project's pr.yaml

jobs:
  shared-vars-setup:
    runs-on: ubuntu-latest
    continue-on-error: true
     env:
        SHARED_WORKFLOWS_REPO: rapidsai/shared-workflows
        SHARED_WORKFLOWS_ACTION: cuda-12.9.0
      steps:
         - name: Stash shared variables setup
           uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main
  cpp-build:
    needs: [shared-vars-setup]
    secrets: inherit
    uses: rapidsai/shared-workflows/.github/workflows/dispatch-conda-cpp-build.yaml@main
    with:
      build_type: ${{ inputs.build_type || 'branch' }}
      branch: ${{ inputs.branch }}

and dispatch-conda-cpp-build.yaml would be:

inputs:
  build_type: ...
  branch: ...

jobs:
  shared-vars-load-or-default:
        steps:
          # Downloads the env var file, loads it, then clones shared-workflows according to env var values
          - name: Load shared variables setup
            uses: rapidsai/shared-actions/load-env-vars-then-clone-workflows@main
   cpp-build:
        needs: [shared-vars-load-or-default]
        uses: ./shared-workflows/.github/workflows/conda-cpp-build.yaml
        with:
            build_type: ${{inputs.build_type}}
            branch: ${{inputs.branch}}

This adds overhead for the indirection, but when there are multiple shared-workflows being called, the developer need only set the values in the base parameter setup:

# This is a project's pr.yaml

jobs:
  shared-vars-setup:
    runs-on: ubuntu-latest
     env:
        SHARED_WORKFLOWS_REPO: rapidsai/shared-workflows
        SHARED_WORKFLOWS_ACTION: cuda-12.9.0
      steps:
        - name: Stash shared variables setup
          uses: rapidsai/shared-actions/telemetry-dispatch-stash-base-env-vars@main

All references to @main or @branch-25.08 in the project's pr.yaml file would not need to change, unless you needed to point to a new or altered dispatch action/workflow.

Implementation note:

Stashing and loading shared variables can utilize the existing shared-actions code. We would need to generalize the clone or copy the script to differentiate destination path of the cloned folder.

[msarahan]

I think this is a minor quality of life improvement. I don't think its priority should be very high, but if you get especially annoyed with search and replace for branch names, you might want to bring this up.

[msarahan]

One other advantage to this that occurred to me: it is recursive. If you have an action or workflow that gets called in some nested way, like:

Workflow A -> action B
workflow A -> workflow C -> action D

Here, if you wanted to test a shared-actions branch (assuming no dispatch actions), you'll have to alter workflow A and workflow C to use the branch for actions B and D. You would need a branch for workflow C, so you'd be coordinating across at least 3 repos. With dispatch actions, you would only need to edit the one branch env var in workflow A. It would obviate the need for the branch on shared-workflows, unless the actual content of shared-workflows was changing.

[jameslamb]

Thanks for writing this up! It's great to have it in writing to discuss and link to.

One clarifying question...

This are written to a file, and stored as a github artifact.

Let's say we implemented something like you're proposing for https://github.com/rapidsai/cudf ... how many additional actions/upload-artifact and actions/download-artifact calls might there be on a single full run of PR CI there?

I ask because we are already somewhat close in some of the more active RAPIDS repos to hitting GitHub's API rate limits, and I believe each call to those actions introduces API calls that count against our limits.

Details on the current state in cudf: https://github.com/rapidsai/build-infra/issues/241#issuecomment-2905841557

Knowing that would be helpful in understanding the possible consequences of adopting something like this.