Skip to content

Commit 7269ac0

Browse files
Updates tests README to mention performance testing
1 parent f34fc32 commit 7269ac0

File tree

1 file changed

+23
-0
lines changed

1 file changed

+23
-0
lines changed

tripy/tests/README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,3 +130,26 @@ Any caption other than `Example` will have a prefix of `Example: ` prepended to
130130

131131
**NOTE: The docstrings must *not* import `tripy`, `numpy`, or `torch`. They will be imported**
132132
**automatically as `tp`, `np`, and `torch` respectively. Any other modules will need to be imported.**
133+
134+
135+
### Performance Tests
136+
137+
In addition to functional tests, we also run performance tests of three kinds:
138+
139+
1. Regression tests, which compare current Tripy performance to historical data
140+
to ensure we don't regress. We use the
141+
[`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/en/latest/)
142+
plugin to gather data and the
143+
[Continuous Benchmark GitHub Action](https://github.com/marketplace/actions/continuous-benchmark)
144+
for regression testing.
145+
146+
You can view graphs and charts of the historical data by opening the
147+
[`index.html` file from the `benchmarks` branch](https://github.com/NVIDIA/TensorRT-Incubator/blob/benchmarks/dev/bench/index.html)
148+
in a browser.
149+
150+
2. Comparative tests, which compare Tripy and `torch.compile`.
151+
152+
3. Overhead tests, which check the overhead introduced by Tripy as compared
153+
to running the underlying MLIR executable by itself. This is done by measuring
154+
how long it takes to run an empty executable since in that case, all the time
155+
is taken by the Tripy wrapper code.

0 commit comments

Comments
 (0)