Add ROCm libraries benchmark tests #2261
Conversation
Utilities with automated system detection, results collection with API integration, and performance tracking with LKG comparison for ROCm libraries. Features: - Benchmark execution with system auto-detection - Local JSON storage and API submission for results - LKG (Last Known Good) comparison - Modular utilities: config, system detection, and results handling - Comprehensive logging with file rotation - Environment variable support for configuration Signed-off-by: Lenine Ajagappane <[email protected]>
Signed-off-by: Lenine Ajagappane <[email protected]>
lajagapp
requested review from
amd-aakash,
gkathirv,
jayhawk-commits and
kiranzangam
lajagapp
changed the title
Signed-off-by: Lenine Ajagappane <[email protected]>
Signed-off-by: Lenine Ajagappane <[email protected]>
Signed-off-by: Lenine Ajagappane <[email protected]>
geomin12
left a comment
geomin12
left a comment
There was a problem hiding this comment.
initial round of comments, there's a lot of code here that may be needed before, but may not be needed now?
also, a few architecture questions i have as well. happy to chat about this via teams or here!
| "hipblaslt_bench": { | ||
| "job_name": "hipblaslt_bench", | ||
| "fetch_artifact_args": "--blas --tests", | ||
| "timeout_minutes": 60, | ||
| "test_script": f"python {_get_script_path('test_hipblaslt_benchmark.py')}", | ||
| # TODO(lajagapp): Add windows test | ||
| "platform": ["linux"], | ||
| "total_shards": 1, | ||
| }, |
There was a problem hiding this comment.
this may be an issue as this will get run during each PR, push to main and scheduled run, probably resulting in long queue times and machine shortages
I would imagine we want to add benchmark tests as a separate (perhaps nightly) run on separate machines? is this the case? how frequent do we want to run these and how long do these take?
| "fetch_artifact_args": "--blas --tests", | ||
| "timeout_minutes": 60, | ||
| "test_script": f"python {_get_script_path('test_hipblaslt_benchmark.py')}", | ||
| # TODO(lajagapp): Add windows test |
There was a problem hiding this comment.
for add windows tests, can we open a Github issue and link it here? so we can keep track?
| ACTIVATION_TYPE = "none" | ||
|
|
||
| # Load benchmark configuration | ||
| config_file = SCRIPT_DIR.parent / 'configs/benchmarks/hipblaslt.json' |
There was a problem hiding this comment.
| config_file = SCRIPT_DIR.parent / 'configs/benchmarks/hipblaslt.json' | |
| config_file = SCRIPT_DIR.parent / 'configs' / 'benchmarks' / 'hipblaslt.json' |
this will make it compatible with windows machines
| # Compare with LKG | ||
| log.info("Comparing results with LKG") | ||
| final_table = client.compare_results(test_name=BENCHMARK_NAME, table=table) | ||
| log.info(f"\n{final_table}") |
There was a problem hiding this comment.
you can also use gha_append_step_summary too to append to the workflow run
| NUM_ITERATIONS = 20 # Number of benchmark iterations | ||
|
|
||
| # Load benchmark configuration | ||
| config_file = SCRIPT_DIR.parent / 'configs/benchmarks/rocfft.json' |
There was a problem hiding this comment.
same here regarding path comment
| """ | ||
| self.api_url = api_url.rstrip('/') | ||
| self.fallback_url = fallback_url.rstrip('/') if fallback_url else None | ||
| self.api_key = api_key |
There was a problem hiding this comment.
secrets in github actions are a bit tricky, particularly for forked PRs (as forked PRs cannot pull data from secrets)
I personally think this should be run a nightly basis as i would imagine it takes quite a while to run (and here, it can pull secrets)
happy to chat about arch here
| """ | ||
| status_code = response.status_code | ||
|
|
||
| if status_code == 401: |
There was a problem hiding this comment.
instead of these elongated error messages, could we just print out the error message given by API? and status code?
| Returns: | ||
| Complete results payload for API submission | ||
| """ | ||
| # Build BM config |
There was a problem hiding this comment.
can we clarify what BM is?
| return payload | ||
|
|
||
|
|
||
| def validate_payload(payload: Dict[str, Any]) -> bool: |
There was a problem hiding this comment.
we may be able to use some python libraries to validate payload :)
| prettytable>=3.0.0 | ||
| requests>=2.28.0 | ||
| jsonschema>=4.0.0 | ||
| packaging>=21.0 |
There was a problem hiding this comment.
can we make the version exact?
Utilities with automated system detection, results collection with API integration, and performance tracking with LKG comparison for ROCm libraries.
Motivation
Add reusable test utilities for ROCm benchmark automation (ROCfft, ROCrand, and more) with standardized system detection, results collection, and LKG performance tracking.
Technical Details
Modular utility package:
Test Plan
Tested benchmark execution, system detection, API submission, and LKG comparison.
Test Result
Benchmarks execute successfully
System detection, API upload, and LKG comparison working correctly
Submission Checklist