Skip to content

Commit 6020dac

Browse files
cerrussellcerrussell
authored
Expand snapshots part I (#1467)
* Extend snapshot testing. Signed-off-by: Caroline Russell <[email protected]> * Remove sdkman step. Signed-off-by: Caroline Russell <[email protected]> * Run java and javascript. Signed-off-by: Caroline Russell <[email protected]> * Troubleshooting. Signed-off-by: Caroline Russell <[email protected]> * Troubleshooting log. Signed-off-by: Caroline Russell <[email protected]> * Enable yarn. Signed-off-by: Caroline Russell <[email protected]> * Bazel troubleshooting. Signed-off-by: Caroline Russell <[email protected]> * Correct bazel build, specify java. Signed-off-by: Caroline Russell <[email protected]> * Set java and bazel env variables. Signed-off-by: Caroline Russell <[email protected]> * Enable skipping specific project. Signed-off-by: Caroline Russell <[email protected]> * Skip jazzer. Signed-off-by: Caroline Russell <[email protected]> * Add sdkman env. Signed-off-by: Caroline Russell <[email protected]> * Troubleshooting sdkman path. Signed-off-by: Caroline Russell <[email protected]> * Troubleshooting sdkman path. Signed-off-by: Caroline Russell <[email protected]> * Troubleshooting sdkman path. Signed-off-by: Caroline Russell <[email protected]> * Correct escapes. Signed-off-by: Caroline Russell <[email protected]> * Revert time logging. Signed-off-by: Caroline Russell <[email protected]> * Repo rename. Signed-off-by: Caroline Russell <[email protected]> * Re-enable prebuild. Signed-off-by: Caroline Russell <[email protected]> * Clean up java prebuilds. Signed-off-by: Caroline Russell <[email protected]> * Correction. Signed-off-by: Caroline Russell <[email protected]> * Remove expanded python and dotnet repos. Signed-off-by: Caroline Russell <[email protected]> * Run all snapshot tests. Signed-off-by: Caroline Russell <[email protected]> * Syntax corrections. Signed-off-by: Caroline Russell <[email protected]> * Locate go. Signed-off-by: Caroline Russell <[email protected]> * Use --migrate-legacy. Signed-off-by: Caroline Russell <[email protected]> * Allow bidirectional --migrate-legacy. Signed-off-by: Caroline Russell <[email protected]> * Remove testing branch checkout. Signed-off-by: Caroline Russell <[email protected]> * Remove self-hosted from matrix. Signed-off-by: Caroline Russell <[email protected]> --------- Signed-off-by: Caroline Russell <[email protected]>
1 parent 5cfb69f commit 6020dac

File tree

4 files changed

+202
-103
lines changed

4 files changed

+202
-103
lines changed

‎.github/workflows/snapshot-tests.yml‎

Lines changed: 26 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ on:
55
pull_request:
66
branches:
77
- master
8+
push:
9+
branches:
10+
- master
811

912

1013
concurrency:
@@ -14,74 +17,55 @@ concurrency:
1417

1518
jobs:
1619

17-
generate:
18-
runs-on: ubuntu-latest
20+
test_non_dotnet:
21+
runs-on: ["snapshot-test"]
1922
steps:
2023

2124
- uses: actions/checkout@v4
2225

23-
- name: Setup Node.js environment
24-
uses: actions/[email protected]
25-
with:
26-
node-version: '23.x'
26+
- name: Set up pnpm
27+
uses: pnpm/action-setup@v4
2728

28-
- name: Setup Python
29-
uses: actions/setup-python@v5
30-
with:
31-
python-version: 3.12
32-
33-
- uses: pnpm/action-setup@v4
34-
35-
- name: cdxgen, sdkman, custom-json-diff installs
29+
- name: cdxgen, custom-json-diff installs
3630
shell: bash
3731
env:
3832
SHELL: bash
3933
run: |
4034
cdxgen_tarball=$(pnpm pack | tail -1)
4135
npm install -g "$cdxgen_tarball"
42-
python -m pip install --upgrade pip
43-
python -m pip install pytest
44-
git clone https://github.com/appthreat/cdxgen-samples.git /home/runner/work/original_snapshots
45-
python -m pip install -r test/diff/requirements.txt
46-
curl -s "https://get.sdkman.io" | bash
47-
source "/home/runner/.sdkman/bin/sdkman-init.sh"
48-
49-
- name: sdkman install cache
50-
uses: actions/cache@v4
51-
id: sdkman-cache
52-
with:
53-
path: /home/runner/.sdkman/candidates
54-
key: "${{ runner.os }}-sdkman-${{ hashFiles('test/diff/repos.csv') }}"
36+
git clone https://github.com/appthreat/cdxgen-samples.git original_snapshots
37+
python3.12 -m venv .venv
38+
source .venv/bin/activate && pip install -r test/diff/requirements.txt
5539
56-
- name: Generate and run scripts
57-
env:
58-
SDKMAN_DIR: /home/runner/.sdkman
59-
CDXGEN_DEBUG_MODE: debug
60-
CDXGEN_LOG: /home/runner/work/new_snapshots/generate.log
61-
PREFER_MAVEN_DEPS_TREE: false
40+
- name: Generate scripts
6241
run: |
63-
mkdir /home/runner/work/new_snapshots
42+
source .venv/bin/activate
6443
python test/diff/generate.py
65-
bash /home/runner/work/new_snapshots/sdkman_installs.sh
66-
bash /home/runner/work/new_snapshots/cdxgen_commands.sh
6744
6845
- name: Upload shell scripts generated as artifact
6946
uses: actions/upload-artifact@v4
7047
with:
7148
name: scripts
72-
path: /home/runner/work/new_snapshots/*.sh
49+
path: new_snapshots/*.sh
50+
51+
- name: Run scripts
52+
env:
53+
PREFER_MAVEN_DEPS_TREE: false
54+
run: |
55+
bash new_snapshots/cdxgen_commands.sh
7356
7457
- name: Upload cdxgen boms
7558
uses: actions/upload-artifact@v4
7659
with:
7760
name: cdxgen_boms
7861
path: |
79-
/home/runner/work/new_snapshots
62+
new_snapshots
8063
8164
- name: Test BOMs
8265
run: |
66+
source .venv/bin/activate
8367
python test/diff/diff_tests.py --migrate-legacy
84-
if test -f /home/runner/work/new_snapshots/diffs.json; then
68+
if test -f new_snapshots/diffs.json; then
8569
echo "status=FAILED" >> "$GITHUB_ENV"
8670
fi
8771
@@ -90,9 +74,9 @@ jobs:
9074
uses: actions/upload-artifact@v4
9175
with:
9276
name: diffs
93-
path: |
94-
/home/runner/work/new_snapshots/diffs.json
95-
/home/runner/work/new_snapshots/*.html
77+
path: |
78+
new_snapshots/diffs.json
79+
new_snapshots/*.html
9680
9781
- name: Exit with error
9882
if: ${{ env.status == 'FAILED' }}

‎test/diff/diff_tests.py‎

Lines changed: 40 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,13 @@
22
import csv
33
import logging
44
import os
5+
from typing import Dict, List, Set
56

67
from custom_json_diff.lib.custom_diff import compare_dicts, perform_bom_diff, report_results
78
from custom_json_diff.lib.custom_diff_classes import Options
8-
from custom_json_diff.lib.utils import json_dump, json_load
9+
from custom_json_diff.lib.utils import json_dump,json_load
10+
11+
from generate import filter_repos
912

1013
logging.disable(logging.INFO)
1114

@@ -15,7 +18,7 @@ def build_args():
1518
parser.add_argument(
1619
'--directories',
1720
'-d',
18-
default=['/home/runner/work/original_snapshots', '/home/runner/work/new_snapshots'],
21+
default=[f'/{os.getenv("GITHUB_WORKSPACE")}/original_snapshots', f'/{os.getenv("GITHUB_WORKSPACE")}/new_snapshots'],
1922
help='Directories containing the snapshots to compare',
2023
nargs=2
2124
)
@@ -25,16 +28,36 @@ def build_args():
2528
action="store_true",
2629
help="Migrate legacy snapshots to 1.6 format"
2730
)
31+
parser.add_argument(
32+
'--projects',
33+
'-p',
34+
help='Filter to these projects.',
35+
dest='projects',
36+
)
37+
parser.add_argument(
38+
'--types',
39+
'-t',
40+
help='Filter to these project types.',
41+
dest='project_types',
42+
)
43+
parser.add_argument(
44+
'--skip-projects',
45+
'-s',
46+
help='Skip these projects'
47+
)
2848
return parser.parse_args()
2949

3050

31-
def compare_snapshot(dir1, dir2, options, repo, migrate_legacy):
51+
def compare_snapshot(dir1: str, dir2: str, options: Options, repo: Dict, migrate_legacy: bool):
3252
bom_1 = f"{dir1}/{repo['project']}-bom.json"
3353
bom_2 = f"{dir2}/{repo['project']}-bom.json"
3454
if migrate_legacy:
3555
bom_data = migrate_to_1_6(bom_1)
3656
bom_1 = bom_1.replace("bom.json", "bom.migrated.json")
3757
json_dump(bom_1, bom_data)
58+
bom_data = migrate_to_1_6(bom_1)
59+
bom_2 = bom_2.replace("bom.json", "bom.migrated.json")
60+
json_dump(bom_2, bom_data)
3861
options.file_1 = bom_1
3962
options.file_2 = bom_2
4063
options.output = f'{dir2}/{repo["project"]}-diff.json'
@@ -45,12 +68,11 @@ def compare_snapshot(dir1, dir2, options, repo, migrate_legacy):
4568
status, result_summary = perform_bom_diff(j1, j2)
4669
report_results(status, result_summary, options, j1, j2)
4770
return status, f"{repo['project']} failed.", result_summary
48-
return status, None, None
49-
71+
return status, f"{repo['project']} succeeded.", {}
5072

51-
def perform_snapshot_tests(dir1, dir2, migrate_legacy):
52-
repo_data = read_csv()
5373

74+
def perform_snapshot_tests(dir1: str, dir2: str, projects: List, project_types: Set, migrate_legacy: bool, skipped_projects):
75+
repo_data = read_csv(projects, project_types, skipped_projects)
5476
options = Options(
5577
allow_new_versions=True,
5678
allow_new_data=True,
@@ -62,11 +84,9 @@ def perform_snapshot_tests(dir1, dir2, migrate_legacy):
6284
failed_diffs = {}
6385
for repo in repo_data:
6486
status, result, summary = compare_snapshot(dir1, dir2, options, repo, migrate_legacy)
65-
if result:
66-
print(result)
87+
print(result)
6788
if status:
6889
failed_diffs[repo["project"]] = summary
69-
7090
if failed_diffs:
7191
diff_file = os.path.join(dir2, 'diffs.json')
7292
print("Snapshot tests failed.")
@@ -94,14 +114,21 @@ def migrate_to_1_6(bom_file):
94114
return bom_data
95115

96116

97-
def read_csv():
117+
def read_csv(projects, project_types, skipped_projects):
98118
csv_file = os.path.join(os.path.dirname(os.path.realpath(__file__)), "repos.csv")
99119
with open(csv_file, 'r', encoding='utf-8') as f:
100120
reader = csv.DictReader(f)
101121
repo_data = list(reader)
102-
return repo_data
122+
return filter_repos(repo_data, projects, project_types, skipped_projects)
103123

104124

105125
if __name__ == '__main__':
106126
args = build_args()
107-
perform_snapshot_tests(args.directories[0], args.directories[1], args.migrate_legacy)
127+
if args.project_types:
128+
if ',' in args.project_types:
129+
project_types = set(args.project_types.split(','))
130+
else:
131+
project_types = {args.project_types}
132+
else:
133+
project_types = set()
134+
perform_snapshot_tests(args.directories[0], args.directories[1], args.projects, project_types, args.migrate_legacy, args.skip_projects)

0 commit comments

Comments
 (0)