Skip to content

Proposed re-write of tools/testharness.py #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Proposed re-write of tools/testharness.py #326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
cc24d4a
Proposed re-write of tools/testharness.py; additional fixes for tests…
Patol75 Jun 12, 2021
cf9bd04
Fixes to testharness based on Actions results; Trying a different YAM…
Patol75 Jun 15, 2021
dfa55e8
Swapping colons for dashes in YAML
Patol75 Jun 15, 2021
37a252b
More colons swaps
Patol75 Jun 15, 2021
d467041
Adding missing square brackets in matrix definitions.
Patol75 Jun 15, 2021
2cb7773
Adding missing checkout step.
Patol75 Jun 15, 2021
ae0d774
Fix missing test_cases when junit_xml is absent; Include missing Pyth…
Patol75 Jun 15, 2021
c0991b0
Miscellaneous fixes; Attempt to only run medium tests.
Patol75 Jun 15, 2021
c59d858
Attempt to run all tests except those that do not complete within 6 h…
Patol75 Jun 17, 2021
b4fb7be
Remove any mention of the special tag, but keep the tests with that t…
Patol75 Jun 23, 2021
59a5ae7
Adjust regex to speed-up pattern matching.
Patol75 Jun 23, 2021
d67342b
Improved diamond_validation; Improved error processing within testhar…
Patol75 Jun 26, 2021
a54c74d
ENH: Merge changes from CMake branch and address reviews.
Patol75 Jun 15, 2022
3d880a5
ENH: Further improvements following review
Patol75 Jun 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
214 changes: 214 additions & 0 deletions .github/workflows/test_suite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
name: Build and Test

on:
push:
branches:
- main

jobs:
build:

name: Build ${{ matrix.release }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
release: [bionic, focal, groovy]

steps:
- name: Check out repository
uses: actions/checkout@v2
- name: Login to Docker Hub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
context: .
file: docker/actions/Dockerfile.actions.${{ matrix.release }}
push: true
tags: fluidity/actions:${{ matrix.release }}-${{ github.sha }}

unittest:

name: Unit tests - ${{ matrix.release }}
runs-on: ubuntu-latest
needs: build

strategy:
fail-fast: false
matrix:
release: [bionic, focal, groovy]
output: [unittests.xml]

steps:
- name: Testing
run: "chmod 777 .

docker pull fluidity/actions:${{ matrix.release }}-${{ github.sha }}

docker run -v $PWD:/host fluidity/actions:${{ matrix.release }}-${{ github.sha }} \
/bin/bash -c \"make unittest && cp -v tests/${{ matrix.output }} \
/host/${{ matrix.release }}-${{ matrix.output }}\""
- name: Upload artifact
uses: actions/upload-artifact@v2
with:
path: ${{ matrix.release }}-${{ matrix.output }}
name: xml_outputs
- name: JUnit report
uses: mikepenz/action-junit-report@v2
with:
report_paths: ${{ matrix.release }}-${{ matrix.output }}
github_token: ${{ secrets.GITHUB_TOKEN }}
check_name: Unit tests report for ${{ matrix.release }}
fail_on_failure: true

shorttest:

name: Short tests - ${{ matrix.release }}
runs-on: ubuntu-latest
needs: build

strategy:
fail-fast: false
matrix:
release: [bionic, focal, groovy]
output: [shorttests.xml]

steps:
- name: Testing
run: "chmod 777 .

docker pull fluidity/actions:${{ matrix.release }}-${{ github.sha }}

docker run -v $PWD:/host fluidity/actions:${{ matrix.release }}-${{ github.sha }} \
/bin/bash -c \"make CORES=2 test && cp -v tests/${{ matrix.output }} \
/host/${{ matrix.release }}-${{ matrix.output }}\""
- name: Upload artifact
uses: actions/upload-artifact@v2
with:
path: ${{ matrix.release }}-${{ matrix.output }}
name: xml_outputs
- name: JUnit report
uses: mikepenz/action-junit-report@v2
with:
report_paths: ${{ matrix.release }}-${{ matrix.output }}
github_token: ${{ secrets.GITHUB_TOKEN }}
check_name: Short tests report for ${{ matrix.release }}
fail_on_failure: true

get-lists:

name: Generate JSON files that contain tests to be executed
runs-on: ubuntu-latest

outputs:
mediumtest-matrix: ${{ steps.set-matrices.outputs.mediumtest-matrix }}
longtest-matrix: ${{ steps.set-matrices.outputs.longtest-matrix }}

strategy:
matrix:
python-version: [3.8]

steps:
- name: Check out repository
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install junit_xml
- name: Generate JSON files
id: set-matrices
# run: "git clone https://github.com/fluidityproject/longtests
run: "tools/testharness.py -l medium --just-list --github mediumtests.json

tools/testharness.py -l long --just-list --github longtests.json

echo \"::set-output name=mediumtest-matrix::$(echo $(cat mediumtests.json))\"

echo \"::set-output name=longtest-matrix::$(echo $(cat longtests.json))\""

mediumtest:

name: Medium test - ${{ matrix.test-xml }} on ${{ matrix.release }}
runs-on: ubuntu-latest
needs: [build, get-lists]

strategy:
fail-fast: false
matrix:
release: [focal]
test-xml: ${{fromJSON(needs.get-lists.outputs.mediumtest-matrix)}}

steps:
- name: Testing
run: "chmod 777 .

docker pull fluidity/actions:${{ matrix.release }}-${{ github.sha }}

docker run -v $PWD:/host fluidity/actions:${{ matrix.release }}-${{ github.sha }} \
/bin/bash -c \"bin/testharness -f ${{ matrix.test-xml }} \
-x ${{ matrix.release }}-${{ matrix.test-xml }} \
&& cp -v ${{ matrix.release }}-${{ matrix.test-xml }} /host\""
- name: Upload artifact
uses: actions/upload-artifact@v2
with:
path: ${{ matrix.release }}-${{ matrix.test-xml }}
name: xml_outputs
- name: JUnit report
uses: mikepenz/action-junit-report@v2
with:
report_paths: ${{ matrix.release }}-${{ matrix.test-xml }}
github_token: ${{ secrets.GITHUB_TOKEN }}
check_name: Report for medium test ${{ matrix.test-xml }} on ${{ matrix.release }}
fail_on_failure: true

longtest:
name: Long test - ${{ matrix.test-xml }} on ${{ matrix.release }}
runs-on: ubuntu-latest
needs: [build, get-lists]

strategy:
fail-fast: false
matrix:
release: [focal]
test-xml: ${{fromJSON(needs.get-lists.outputs.longtest-matrix)}}

steps:
- name: Testing
# run: "chmod 777 .
#
# docker pull fluidity/actions:${{ matrix.release }}-${{ github.sha }}
#
# docker run -v $PWD:/host fluidity/actions:${{ matrix.release }}-${{ github.sha }} \
# /bin/bash -c \"git clone https://github.com/fluidityproject/longtests \
# && bin/testharness -l long -f ${{ matrix.test-xml }} \
# -x ${{ matrix.release }}-${{ matrix.test-xml }} \
# && cp -v ${{ matrix.release }}-${{ matrix.test-xml }} /host\""
run: "chmod 777 .

docker pull fluidity/actions:${{ matrix.release }}-${{ github.sha }}

docker run -v $PWD:/host fluidity/actions:${{ matrix.release }}-${{ github.sha }} \
/bin/bash -c \"bin/testharness -l long -f ${{ matrix.test-xml }} \
-x ${{ matrix.release }}-${{ matrix.test-xml }} \
&& cp -v ${{ matrix.release }}-${{ matrix.test-xml }} /host\""
- name: Upload artifact
uses: actions/upload-artifact@v2
with:
path: ${{ matrix.release }}-${{ matrix.test-xml }}
name: xml_outputs
- name: JUnit report
uses: mikepenz/action-junit-report@v2
with:
report_paths: ${{ matrix.release }}-${{ matrix.test-xml }}
github_token: ${{ secrets.GITHUB_TOKEN }}
check_name: Report for long test ${{ matrix.test-xml }} on ${{ matrix.release }}
fail_on_failure: true
Loading