-
Notifications
You must be signed in to change notification settings - Fork 12
docs: adding documentation for the BSEQ and WIT benchmarks #606
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
vprusso
commented
Oct 28, 2025
- Adding documentation pages for each of the (currently) supported benchmarks in metriq-gym.
- Added specific content for both the BSEQ and WIT benchmarks (but deliberately kept the other benchmarks sparse and relatively empty for now, as I am not sure if this is the way in which we want to document/capture these details). That is, there might be more granular things we wish to cover or even cover it in a completely different place (like the metriq website, for instance). Keeping sparse for now until we refine our approach to what we want/need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces comprehensive documentation for benchmarks supported by metriq-gym. The documentation provides users with reference material for understanding and executing quantum benchmarks, including detailed specifications for BSEQ and WIT benchmarks while establishing placeholder pages for other benchmarks pending future documentation decisions.
Key Changes:
- Added a new "Benchmark Reference" section to the documentation with dedicated pages for 11 benchmarks
- Provided complete documentation for BSEQ and WIT benchmarks including overview, schema parameters, usage instructions, and result interpretation
- Created placeholder documentation pages for 9 additional benchmarks with basic structure and usage hints
Reviewed Changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| docs/source/index.rst | Added link to new benchmark reference section in the documentation table of contents |
| docs/source/benchmarks/index.rst | Created benchmark reference index page listing all 11 supported benchmarks |
| docs/source/benchmarks/bseq.rst | Full documentation for BSEQ benchmark including parameters, usage, and result interpretation |
| docs/source/benchmarks/wit.rst | Full documentation for WIT benchmark including parameters, usage, and result interpretation |
| docs/source/benchmarks/clops.rst | Placeholder documentation page for CLoPS benchmark |
| docs/source/benchmarks/quantum_volume.rst | Placeholder documentation page for Quantum Volume benchmark |
| docs/source/benchmarks/mirror_circuits.rst | Placeholder documentation page for Mirror Circuits benchmark |
| docs/source/benchmarks/qml_kernel.rst | Placeholder documentation page for QML Kernel benchmark |
| docs/source/benchmarks/lr_qaoa.rst | Placeholder documentation page for Linear Ramp QAOA benchmark |
| docs/source/benchmarks/bernstein_vazirani.rst | Placeholder documentation page for Bernstein-Vazirani benchmark |
| docs/source/benchmarks/phase_estimation.rst | Placeholder documentation page for Phase Estimation benchmark |
| docs/source/benchmarks/hidden_shift.rst | Placeholder documentation page for Hidden Shift benchmark |
| docs/source/benchmarks/quantum_fourier_transform.rst | Placeholder documentation page for Quantum Fourier Transform benchmark |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Let's add the |
cosenal
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, let's remove QV and CLOPS docstrings, as we are not supporting those benchmarks at the moment.
metriq_gym/benchmarks/wit.py
Outdated
| - num_qubits (int, optional, default 6): choose 6 or 7 depending on hardware topology; 7 | ||
| aligns with Fig. 4 of the reference paper. | ||
| CLI dispatch example:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need a dispatch example here, it's going to be the same for all the benchmarks anyway
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| then invert them to test how well a device preserves state fidelity across the forward | ||
| and reverse halves of the circuit. | ||
| Schema parameters (metriq_gym/schemas/mirror_circuits.schema.json): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here I am afraid we are duplicating information with the one we already have in the schema files. In fact, we already have description in all fields in https://github.com/unitaryfoundation/metriq-gym/blob/main/metriq_gym/schemas/mirror_circuits.schema.json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.