Skip to content
This repository was archived by the owner on May 9, 2024. It is now read-only.

[CI] Enable Conda cache #453

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[CI] Enable Conda cache #453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions .github/workflows/build-windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,21 +43,30 @@ jobs:
Get-PSDrive

- name: Restore Conda env cache
id: conda-cache
uses: actions/cache@v3
id: restore-conda-cache
uses: actions/cache/restore@v3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, better to have the same action name in all places; if you add "restore" here, add everywhere

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ env.RUN_STAMP }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-windows-env.yml') }}-${{ env.DATE }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-windows-env.yml') }}-${{ env.DATE }}
restore-keys: |
${{ env.RUN_STAMP }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-windows-env.yml') }}-
${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-windows-env.yml') }}-

- name: Update Conda env
if: steps.conda-cache.cache-hit != 'true'
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
run: |
& $env:CONDA\condabin\conda.bat update conda
& $env:CONDA\condabin\conda.bat env update -f omniscidb/scripts/mapd-deps-conda-windows-env.yml

- name: Save Conda env cache
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}


- name: Restore Maven cache
uses: actions/cache@v3
with:
Expand Down
18 changes: 13 additions & 5 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,21 +44,29 @@ jobs:
echo "DATE=$(date +'%Y%m%d')" >> $GITHUB_ENV

- name: Restore Conda env cache
id: conda-cache
uses: actions/cache@v3
id: restore-conda-cache
uses: actions/cache/restore@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ env.RUN_STAMP }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
restore-keys: |
${{ env.RUN_STAMP }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-
${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-

- name: Update Conda env
if: steps.conda-cache.cache-hit != 'true'
if: steps.restore-conda-cacheoutputs.cache-hit != 'true'
run: |
conda update conda
conda env update -f omniscidb/scripts/mapd-deps-conda-dev-env.yml

- name: Save Conda env cache
if: steps.restore-conda-cacheoutputs.cache-hit != 'true'
uses: actions/cache/save@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}

- name: Install cuda
if: inputs.name == 'cuda' || inputs.name == 'all-gpus'
run: |
Expand Down
23 changes: 21 additions & 2 deletions .github/workflows/modin.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,33 @@ jobs:

- name: Set env context
run: |
echo RUN_STAMP=${{ runner.os }}-${{ (env.cuda_compiler_version != 'None') && format('cuda{0}', env.cuda_compiler_version) || 'cpu' }} >>$GITHUB_ENV
echo RUN_STAMP=${{ runner.os }}-cpu >>$GITHUB_ENV
echo CONDA_ENV_PATH=$CONDA/envs/${{ env.CONDA_ENV }} >>$GITHUB_ENV

- name: Update Conda
- name: Restore Conda env cache
id: restore-conda-cache
uses: actions/cache/restore@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
restore-keys: |
${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-

- name: Update Conda env
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
run: |
conda update conda
conda env update -f omniscidb/scripts/mapd-deps-conda-dev-env.yml

- name: Save Conda env cache
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}

- name: Restore Maven cache
uses: actions/cache@v3
with:
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/pytest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- name: Set env context
run: |
echo CONDA_PATH=$CONDA >>$GITHUB_ENV
echo RUN_STAMP=${{ runner.os }}-${{ (env.cuda_compiler_version != 'None') && format('cuda{0}', env.cuda_compiler_version) || 'cpu' }} >>$GITHUB_ENV
echo RUN_STAMP=${{ runner.os }}-cpu >>$GITHUB_ENV
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both pytest.yml and modin.yml should be made a part of test.yml, thus the whole "Set env context" section will be reusable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will change it later in further prs

echo CONDA_ENV_PATH=$CONDA/envs/${{ env.CONDA_ENV }} >>$GITHUB_ENV

- name: Restore Conda env cache
Expand All @@ -25,23 +25,23 @@ jobs:
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ env.RUN_STAMP }}-conda-pytest-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
restore-keys: |
${{ env.RUN_STAMP }}-conda-pytest-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-
${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-

- name: Update Conda env
if: steps.conda-cache.cache-hit != 'true'
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
run: |
conda update conda
conda env update -f omniscidb/scripts/mapd-deps-conda-dev-env.yml

- name: Save Conda env cache
id: save-conda-cache
if: steps.restore-conda-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v3
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ env.RUN_STAMP }}-conda-pytest-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ env.DATE }}

- name: Restore Maven cache
uses: actions/cache@v3
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
with:
path: |
${{ env.CONDA_ENV_PATH }}
key: ${{ env.RUN_STAMP }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ inputs.cache-suffix }}
key: ${{ runner.os }}-conda-${{ hashFiles('omniscidb/scripts/mapd-deps-conda-dev-env.yml') }}-${{ inputs.cache-suffix }}

- name: C++ style check
if: inputs.test == 'style'
Expand Down