Skip to content

Conversation

@vprusso
Copy link
Contributor

@vprusso vprusso commented Oct 28, 2025

  • Adding documentation pages for each of the (currently) supported benchmarks in metriq-gym.
  • Added specific content for both the BSEQ and WIT benchmarks (but deliberately kept the other benchmarks sparse and relatively empty for now, as I am not sure if this is the way in which we want to document/capture these details). That is, there might be more granular things we wish to cover or even cover it in a completely different place (like the metriq website, for instance). Keeping sparse for now until we refine our approach to what we want/need.

Copilot AI review requested due to automatic review settings October 28, 2025 19:08
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces comprehensive documentation for benchmarks supported by metriq-gym. The documentation provides users with reference material for understanding and executing quantum benchmarks, including detailed specifications for BSEQ and WIT benchmarks while establishing placeholder pages for other benchmarks pending future documentation decisions.

Key Changes:

  • Added a new "Benchmark Reference" section to the documentation with dedicated pages for 11 benchmarks
  • Provided complete documentation for BSEQ and WIT benchmarks including overview, schema parameters, usage instructions, and result interpretation
  • Created placeholder documentation pages for 9 additional benchmarks with basic structure and usage hints

Reviewed Changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated no comments.

Show a summary per file
File Description
docs/source/index.rst Added link to new benchmark reference section in the documentation table of contents
docs/source/benchmarks/index.rst Created benchmark reference index page listing all 11 supported benchmarks
docs/source/benchmarks/bseq.rst Full documentation for BSEQ benchmark including parameters, usage, and result interpretation
docs/source/benchmarks/wit.rst Full documentation for WIT benchmark including parameters, usage, and result interpretation
docs/source/benchmarks/clops.rst Placeholder documentation page for CLoPS benchmark
docs/source/benchmarks/quantum_volume.rst Placeholder documentation page for Quantum Volume benchmark
docs/source/benchmarks/mirror_circuits.rst Placeholder documentation page for Mirror Circuits benchmark
docs/source/benchmarks/qml_kernel.rst Placeholder documentation page for QML Kernel benchmark
docs/source/benchmarks/lr_qaoa.rst Placeholder documentation page for Linear Ramp QAOA benchmark
docs/source/benchmarks/bernstein_vazirani.rst Placeholder documentation page for Bernstein-Vazirani benchmark
docs/source/benchmarks/phase_estimation.rst Placeholder documentation page for Phase Estimation benchmark
docs/source/benchmarks/hidden_shift.rst Placeholder documentation page for Hidden Shift benchmark
docs/source/benchmarks/quantum_fourier_transform.rst Placeholder documentation page for Quantum Fourier Transform benchmark

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cosenal
Copy link
Contributor

cosenal commented Oct 29, 2025

Let's add the benchmarks folder in the list of automatically generated API code here: https://github.com/unitaryfoundation/metriq-gym/blob/main/docs/source/api.rst

@vprusso vprusso changed the title docs: adding documentation for the benchmarks docs: adding documentation for the BSEQ and WIT benchmarks Nov 7, 2025
Copy link
Contributor

@cosenal cosenal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, let's remove QV and CLOPS docstrings, as we are not supporting those benchmarks at the moment.

- num_qubits (int, optional, default 6): choose 6 or 7 depending on hardware topology; 7
aligns with Fig. 4 of the reference paper.
CLI dispatch example::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need a dispatch example here, it's going to be the same for all the benchmarks anyway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

then invert them to test how well a device preserves state fidelity across the forward
and reverse halves of the circuit.
Schema parameters (metriq_gym/schemas/mirror_circuits.schema.json):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I am afraid we are duplicating information with the one we already have in the schema files. In fact, we already have description in all fields in https://github.com/unitaryfoundation/metriq-gym/blob/main/metriq_gym/schemas/mirror_circuits.schema.json

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@vprusso vprusso requested review from Copilot and cosenal November 25, 2025 17:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants