diff --git a/README.md b/README.md index f011ed3..9b61358 100644 --- a/README.md +++ b/README.md @@ -1,315 +1,84 @@ - - Gitmoji - + -Increase verbosity by repeating the `-v` / `--verbose` flag: +

+ + + + +

-- No `-v` → WARNING (default) -- `-v` → INFO -- `-vv` → DEBUG - -Examples: -```bash -mqssbench -v run --config path/to/config.yaml -mqssbench -vv run --config path/to/config.yaml -``` - -Benchmark results are printed to standard output, while logs are sent to standard error. - -### Configuration file format - -A typical benchmark configuration file follows the structure shown below: - -```yaml -# Benchmark configuration template - -# Specify the benchmark to run. Two formats are supported: -# 1. String format: "origin/source/name" -# 2. Structured format: -# origin: -# source: -# name: -benchmark: // - -# Benchmark specific parameters -benchmark_params: - : - : - : - -# Adapter configuration -adapter: -adapter_params: - : - : - : - -# output directory -output_dir: - -# Controls what is generated (analysis, visualizations, reports) -report: - analysis: - enabled: - visualization: - enabled: - show: - -# Controls how results are persisted -storage: - enabled: - type: # "file" or "sqlite" - file: - format: #"json" - sqlite: # this will be implemeted in future - db_path: - -# Profiling controls -profiling: - enabled: - metrics: - # List of profiling metrics. If omitted, all supported metrics are collected. - # Available metrics depend on your selected adaptor. - # For example, for MQSS adaptors, these are valid metrics: - # mqp_api, quantum_database, quantum_job_runner, isv_job_runner, - # quantum_daemon_job_runner, generator, scheduler, pass_runner, - # transpiler, submitter, pass_selection, knitter, job_execution - - - - -``` - -Example of a Single benchmark config: -```yaml -benchmark: core/native/randomized_benchmarking - -benchmark_params: - num_qubits: 2 - lengths: [2, 4, 8, 16] - num_sequences: 2 - -adapter: mqss_qiskit -adapter_params: - backend: QExa20 - backend_params: - queued: true - credentials: - mqss_token: "" - shots: 1000 - -output_dir: "./results" - -report: - analysis: - enabled: true - visualization: - enabled: true - show: false - -storage: - enabled: true - type: "file" - file: - format: "json" - -profiling: - enabled: true - metrics: - - transpiler - - submitter -``` - - -Example of a multi benchmark config: - -```yaml -- benchmark: core/native/randomized_benchmarking - benchmark_params: - num_qubits: 2 - lengths: [2, 4, 8] - num_sequences: 2 - adapter: mqss_qiskit - adapter_params: - backend: QExa20 - credentials: - mqss_token: "" - shots: 200 - output_dir: "./results" - storage: - enabled: true - type: "file" - file: - format: "json" - -- benchmark: core/native/quantum_volume - benchmark_params: - num_qubits: 3 - depth: 3 - trials: 2 - adapter: mqss_qiskit - adapter_params: - backend: QExa20 - credentials: - mqss_token: "" - shots: 200 - output_dir: "./results" - storage: - enabled: true - type: "file" - file: - format: "json" -``` - -## 🧩 Python API Usage +# MQSS Benchmarking Framework -MQSS Benchmarking Framework can also be used directly as a Python library when more control is needed. +

+ + Documentation + +

-```python -from mqssbench.runtime import BenchmarkManager +The **MQSS Benchmarking Framework** is a general-purpose, platform-agnostic, extensible quantum benchmarking framework designed to serve as the central orchestration and integration layer for the quantum benchmarking ecosystem. It provides a unified infrastructure that brings together benchmarks, hardware backends, simulators, circuit providers, benchmark suites, and benchmarking frameworks under a single, consistent environment, covering the full spectrum of benchmarking experiments. -list_of_benchmarks = BenchmarkManager.get_available_benchmarks() +MQSS Benchmarking Framework is built around a principled architecture based on a registry-driven component system, abstract interfaces for extensible components, and a plugin discovery mechanism. This design allows benchmarks, hardware backends, and tools to be integrated as first-class components within the framework. The framework is under active development, with an initial core library already in place and more components planned as the architecture evolves. -config = { - "benchmark": "core/native/randomized_benchmarking", - "benchmark_params": { - "num_qubits": 2, - "lengths": [2, 4, 8, 16], - "num_sequences": 2 - }, +## FAQ - "adapter": "mqss_qiskit", - "adapter_params": { - "backend": "QExa20", - "credentials": { - "mqss_token": "" - }, - "shots": 1000 - }, +### What is MQSS? - "output_dir": "./results", +_MQSS_ stands for _Munich Quantum Software Stack_ and is a project of the _Munich Quantum Valley_ +initiative. It is jointly developed by the _Munich Quantum Valley (MQV) gGmbH_, _Leibniz +Supercomputing Centre (LRZ)_, the _Chair for Design Automation (CDA)_, and the _Chair of Computer +Architecture and Parallel Systems (CAPS)_ at TUM. It provides a comprehensive compilation and +runtime infrastructure for on-premise and remote quantum devices, support for modern compilation and +optimization techniques, and enables both current and future high-level abstractions for quantum +programming. This stack is designed to be capable of deployment in a variety of scenarios via +flexible configuration options. This includes stand-alone scenarios for individual systems, cloud +access to a variety of devices, as well as tight integration into HPC environments supporting +quantum acceleration. Concrete instances of the _MQSS_ are deployed at the LRZ and MQV gGmbH, +providing unified access to all of their quantum devices through multiple compatible access paths. +This includes a web portal, command line access via web credentials, as well as the option for +hybrid access with tight integration with HPC systems.It facilitates the connection between +end-users and quantum computing platforms by its integration within HPC infrastructures, such as +those found at the LRZ. - "report": { - "analysis": { - "enabled": True, - "visualization": { - "enabled": True - } - } - }, +### Where can I find the getting started guide? - "storage": { - "enabled": True, - "type": "file", - "file": { - "format": "json" - } - }, - - "profiling": { - "enabled": True, - "metrics": ["transpiler", "submitter"] - } -} +A step-by-step guide is available in the documentation site at [Getting Started](docs/user_guide/getting_started.md). -benchmark_manager = BenchmarkManager(config) -benchmark_manager.dispatch() -``` +### How can I contribute to the project? -The `benchmark` field must always follow the strict `origin/source/name` format. For example: +Contribution guidelines are available in [CONTRIBUTING.md](CONTRIBUTING.md). -- Core native benchmark: `core/native/quantum_volume` - uses `native` (native provider) as source -- Core provider benchmark: `core/mqt_bench/vqe_su2` - uses `mqt_bench` (a circuit provider) as source -- User defined benchmark: `user/my_source/my_benchmark_name` — uses `my_source` (arbitrary user source), can also leverage circuit providers -Both CLI and Python API share the same execution engine, registry system, and adapter logic. +### Under which license is MQSS Benchmarking Framework released? -### Latest supported adapters -- Qiskit Adapter (config value `mqss_qiskit`) -- Pennylane Adapter (config value `mqss_pennylane`) +**MQSS Benchmarking Framework** is released under the Apache License v2.0. See +[LICENSE](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/blob/develop/LICENSE.md) for +more information. Any contribution to the project is assumed to be under the same license. -A complete list of configuration options will be listed and constantly updated for the upcoming releases +## Contact -## 🛠️ Upcoming Features -- Integration of the [Toolchain Project]([https://pages.github.com/](https://gitlab.lrz.de/qcbm/benchmark-comparison)) to the Framework for Simulator Benchmarks -- Improving the benchmark set for all types of benchmarks -## 📝 Contributing +The development of this project is led by the QCT department at the LRZ and the QSI department at +MQV gGmbH. You can also always reach us at +[mqss@munich-quantum-valley.de](mailto:mqss@munich-quantum-valley.de). -Feel free to open issues or submit pull requests to improve this project! +Please try to use the publicly accessible GitHub channels +([issues](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/issues), +[discussions](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/discussions), +[pull requests](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/pulls)) to allow for a +transparent and open discussion as much as possible. diff --git a/docs/_static/mqss_logo.svg b/docs/_static/mqss_logo.svg new file mode 100644 index 0000000..b2089b7 --- /dev/null +++ b/docs/_static/mqss_logo.svg @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/_static/mqss_logo_dark.svg b/docs/_static/mqss_logo_dark.svg new file mode 100644 index 0000000..8ab2fe5 --- /dev/null +++ b/docs/_static/mqss_logo_dark.svg @@ -0,0 +1,240 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/_static/runtime_execution_flow.png b/docs/_static/runtime_execution_flow.png new file mode 100644 index 0000000..082de71 Binary files /dev/null and b/docs/_static/runtime_execution_flow.png differ diff --git a/docs/_static/system_architecture.png b/docs/_static/system_architecture.png deleted file mode 100644 index 6da4dd8..0000000 Binary files a/docs/_static/system_architecture.png and /dev/null differ diff --git a/docs/development_guide/architecture_overview.md b/docs/development_guide/architecture_overview.md new file mode 100644 index 0000000..c5231e2 --- /dev/null +++ b/docs/development_guide/architecture_overview.md @@ -0,0 +1,88 @@ +# Architecture Overview + +The **MQSS Benchmarking Framework** is organized as an extensible orchestration layer for quantum benchmarking workflows. + +> **Note** +> This document provides a high-level overview of the current MQSS Benchmarking Framework architecture. The framework is under active development, and additional architectural details, diagrams, and component documentation may be added as the project evolves. + +## Design goals + +- Support multiple benchmark types +- Integrate hardware backends and simulators consistently +- Keep components modular and replaceable +- Enable easy extension via new benchmarks, suites, and plugins + +## Core ideas + +### Registry-based component model + +MQSS Benchmarking Framework uses a registry-driven system where all components are registered, discovered, and instantiated dynamically at runtime. + +### Abstract interfaces + +All framework components communicate through abstract interfaces rather than direct implementations. This ensures modularity and extensibility. + +### Plugin discovery + +MQSS Benchmarking Framework supports a plugin-based architecture so new benchmarking components can be added without modifying the core framework. + +## Main components + +- **Benchmarks**: define what is measured and evaluated +- **Device adapters**: interface with hardware backends and simulators +- **Circuit providers**: supply circuits or workload definitions +- **Framework core**: handles discovery, setup, orchestration, and execution flow + +## Execution flow + +1. A benchmark is selected via CLI with config file. +2. Required components are resolved through the registry. +3. Backend or simulator is selected via device adapter. +4. Benchmark execution is performed. +5. Results are collected and processed into metrics and plots. +6. Final results are stored or returned to the user. + +### Execution Flow Diagram + +```mermaid +flowchart TD + + subgraph Init["Initialization"] + INIT["Initialize System"] + REG["Auto Registration"] + MODS["Built-in Modules"] + REGISTRY["Central Registry"] + + INIT --> REG --> MODS --> REGISTRY + end + + CLI["CLI Entry"] --> MANAGER["Benchmark Manager"] + + subgraph Commands["Command Handling"] + MANAGER --> LIST["List Commands"] + LIST --> REGISTRY + + MANAGER --> RUNNER["Benchmark Runner"] + end + + RUNNER --> CONFIG["Parse Config"] + CONFIG --> REGISTRY + + REGISTRY --> COMPONENTS["Resolve Components"] + COMPONENTS --> BENCH["Benchmark"] + + subgraph Core["Benchmark Execution"] + BENCH --> GEN["Circuit Generator"] + GEN --> EXEC["Executor"] + EXEC --> RESULTS["Raw Results"] + RESULTS --> ANALYZER["Result Analyzer"] + ANALYZER --> METRICS["Metrics Output"] + end + + METRICS --> RETURN["Return Results"] + RETURN --> STORAGE["Storage (Optional)"] +``` + +--- + +← [Back to Documentation Home](../index.md) \ No newline at end of file diff --git a/docs/development_guide/contributing.md b/docs/development_guide/contributing.md new file mode 100644 index 0000000..6c7b191 --- /dev/null +++ b/docs/development_guide/contributing.md @@ -0,0 +1,15 @@ +# Contributing Guide + +This page mirrors the contribution instructions in the root [CONTRIBUTING.md](../../CONTRIBUTING.md). + +Please read the root contribution guide for the full workflow, conventions, and development process. + +## Quick Links + +- [Repository](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework) +- [Repository issues](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/issues) +- [Pull requests](https://github.com/Munich-Quantum-Software-Stack/MQSS-Benchmarking-Framework/pulls) + +--- + +← [Back to Documentation Home](../index.md) \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..a0da808 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,19 @@ +# MQSS Benchmarking Framework + +The **MQSS Benchmarking Framework** is a general-purpose, platform-agnostic, extensible quantum benchmarking framework designed to serve as the central orchestration and integration layer for the quantum benchmarking ecosystem. It provides a unified infrastructure that brings together benchmarks, hardware backends, simulators, circuit providers, benchmark suites, and benchmarking frameworks under a single, consistent environment, covering the full spectrum of benchmarking experiments. + +MQSS Benchmarking Framework is built around a principled architecture based on a registry-driven component system, abstract interfaces for extensible components, and a plugin discovery mechanism. This design allows benchmarks, hardware backends, and tools to be integrated as first-class components within the framework. The framework is under active development, with an initial core library already in place and more components planned as the architecture evolves. + +## What this documentation contains + +This documentation provides user guides, developer documentation, and technical information about the MQSS Benchmarking Framework. + +## Quick Links + +- [Getting Started](user_guide/getting_started.md) +- [Architecture Overview](development_guide/architecture_overview.md) +- [Contributing Guide](development_guide/contributing.md) + +## API Reference + +*Coming soon.* \ No newline at end of file diff --git a/docs/system_architecture.md b/docs/system_architecture.md deleted file mode 100644 index 784506f..0000000 --- a/docs/system_architecture.md +++ /dev/null @@ -1,67 +0,0 @@ -# System Architecture Diagram -```mermaid -flowchart TD - - %% ----------------------- - %% Initialization (Side) - %% ----------------------- - subgraph Init["Initialization"] - INIT["Initialize System"] - REG["Auto Registration"] - MODS["Built-in Modules"] - REGISTRY["Central Registry"] - - INIT --> REG --> MODS --> REGISTRY - end - - - %% ----------------------- - %% Entry - %% ----------------------- - CLI["CLI Entry (main.py)"] --> MANAGER["Benchmark Manager"] - - - %% ----------------------- - %% Command Handling - %% ----------------------- - subgraph Commands["Command Handling"] - - %% Info / Listing Commands - MANAGER --> LIST["List Commands"] - LIST --> REGISTRY - - %% Execution Commands - MANAGER --> RUNNER["Benchmark Runner"] - end - - - %% ----------------------- - %% Execution Flow - %% ----------------------- - RUNNER --> CONFIG["Parse Config"] - CONFIG --> REGISTRY - - REGISTRY --> COMPONENTS["Resolve Components"] - COMPONENTS --> BENCH["Benchmark"] - - - %% ----------------------- - %% Benchmark Execution Core - %% ----------------------- - subgraph Core["Benchmark Execution"] - - BENCH --> GEN["Circuit Generator"] - GEN --> EXEC["Executor"] - EXEC --> RESULTS["Raw Results"] - RESULTS --> ANALYZER["Result Analyzer"] - ANALYZER --> METRICS["Metrics Output"] - - end - - - %% ----------------------- - %% Finalization - %% ----------------------- - METRICS --> RETURN["Return Results"] - RETURN --> STORAGE["Storage (Optional)"] -``` diff --git a/docs/user_guide/getting_started.md b/docs/user_guide/getting_started.md new file mode 100644 index 0000000..27deb43 --- /dev/null +++ b/docs/user_guide/getting_started.md @@ -0,0 +1,316 @@ + + Gitmoji + + +# MQSS Benchmarking Framework + +## Overview + +MQSS Benchmarking Framework is an automated and reproducable tool for uniting Quantum Computing Benchmarks. It has 4 main pillars: + +- Hardware Benchmarks +- Software Benchmarks +- Simulator Benchmarks +- Algorithmic Benchmarks + +## 🛠️ Installation + +This project leverages [uv](https://github.com/astral-sh/uv) for dependency management and reproducibility, making setup and collaboration straightforward. +To get started, ensure you have Python installed. Then, follow these steps to set up your environment using `uv`: + +1. **Install uv** (if you don't have it already): + + ```bash + pip install uv + ``` + +2. **Sync your environment with the project's dependencies:** + + ```bash + uv sync + ``` + +## 🚀 Usage + +MQSS Benchmarking Framework can be used in two ways: through a simple command-line interface or directly as a Python library. + +## 💻 CLI Usage + +After installation, the command `mqssbench` becomes available system-wide. + +### Listing benchmarks + +```bash +mqssbench list +``` + +This prints every benchmark registered under the `origin/source/name` structure, including internal, external and user-defined benchmarks. + +### Running benchmarks + +```bash +mqssbench run --config path/to/config.yaml +``` + +The `--config` file may contain either: +1. a single benchmark configuration (dict) +2. or a list of benchmark configurations (each a full config dict) + +When a list is provided, mqssbench runs each benchmark sequentially using the same execution engine. + +In development setups, you can run the CLI commands through `uv`: + +```bash +uv run mqssbench list +uv run mqssbench run --config path/to/config.yaml +``` + +To explore all available commands and options: + +```bash +mqssbench --help +``` + +### Verbosity and logging + +The CLI supports adjustable logging verbosity for debugging and inspection. + +By default, only warnings and errors are shown. + +Increase verbosity by repeating the `-v` / `--verbose` flag: + +- No `-v` → WARNING (default) +- `-v` → INFO +- `-vv` → DEBUG + +Examples: +```bash +mqssbench -v run --config path/to/config.yaml +mqssbench -vv run --config path/to/config.yaml +``` + +Benchmark results are printed to standard output, while logs are sent to standard error. + +### Configuration file format + +A typical benchmark configuration file follows the structure shown below: + +```yaml +# Benchmark configuration template + +# Specify the benchmark to run. Two formats are supported: +# 1. String format: "origin/source/name" +# 2. Structured format: +# origin: +# source: +# name: +benchmark: // + +# Benchmark specific parameters +benchmark_params: + : + : + : + +# Adapter configuration +adapter: +adapter_params: + : + : + : + +# output directory +output_dir: + +# Controls what is generated (analysis, visualizations, reports) +report: + analysis: + enabled: + visualization: + enabled: + show: + +# Controls how results are persisted +storage: + enabled: + type: # "file" or "sqlite" + file: + format: #"json" + sqlite: # this will be implemeted in future + db_path: + +# Profiling controls +profiling: + enabled: + metrics: + # List of profiling metrics. If omitted, all supported metrics are collected. + # Available metrics depend on your selected adaptor. + # For example, for MQSS adaptors, these are valid metrics: + # mqp_api, quantum_database, quantum_job_runner, isv_job_runner, + # quantum_daemon_job_runner, generator, scheduler, pass_runner, + # transpiler, submitter, pass_selection, knitter, job_execution + - + - +``` + +Example of a Single benchmark config: +```yaml +benchmark: core/native/randomized_benchmarking + +benchmark_params: + num_qubits: 2 + lengths: [2, 4, 8, 16] + num_sequences: 2 + +adapter: mqss_qiskit +adapter_params: + backend: QExa20 + backend_params: + queued: true + credentials: + mqss_token: "" + shots: 1000 + +output_dir: "./results" + +report: + analysis: + enabled: true + visualization: + enabled: true + show: false + +storage: + enabled: true + type: "file" + file: + format: "json" + +profiling: + enabled: true + metrics: + - transpiler + - submitter +``` + + +Example of a multi benchmark config: + +```yaml +- benchmark: core/native/randomized_benchmarking + benchmark_params: + num_qubits: 2 + lengths: [2, 4, 8] + num_sequences: 2 + adapter: mqss_qiskit + adapter_params: + backend: QExa20 + credentials: + mqss_token: "" + shots: 200 + output_dir: "./results" + storage: + enabled: true + type: "file" + file: + format: "json" + +- benchmark: core/native/quantum_volume + benchmark_params: + num_qubits: 3 + depth: 3 + trials: 2 + adapter: mqss_qiskit + adapter_params: + backend: QExa20 + credentials: + mqss_token: "" + shots: 200 + output_dir: "./results" + storage: + enabled: true + type: "file" + file: + format: "json" +``` + +## 🧩 Python API Usage + +MQSS Benchmarking Framework can also be used directly as a Python library when more control is needed. + +```python +from mqssbench.runtime import BenchmarkManager + +list_of_benchmarks = BenchmarkManager.get_available_benchmarks() + +config = { + "benchmark": "core/native/randomized_benchmarking", + + "benchmark_params": { + "num_qubits": 2, + "lengths": [2, 4, 8, 16], + "num_sequences": 2 + }, + + "adapter": "mqss_qiskit", + "adapter_params": { + "backend": "QExa20", + "credentials": { + "mqss_token": "" + }, + "shots": 1000 + }, + + "output_dir": "./results", + + "report": { + "analysis": { + "enabled": True, + "visualization": { + "enabled": True + } + } + }, + + "storage": { + "enabled": True, + "type": "file", + "file": { + "format": "json" + } + }, + + "profiling": { + "enabled": True, + "metrics": ["transpiler", "submitter"] + } +} + +benchmark_manager = BenchmarkManager(config) +benchmark_manager.dispatch() +``` + +The `benchmark` field must always follow the strict `origin/source/name` format. For example: + +- Core native benchmark: `core/native/quantum_volume` - uses `native` (native provider) as source +- Core provider benchmark: `core/mqt_bench/vqe_su2` - uses `mqt_bench` (a circuit provider) as source +- User defined benchmark: `user/my_source/my_benchmark_name` — uses `my_source` (arbitrary user source), can also leverage circuit providers + +Both CLI and Python API share the same execution engine, registry system, and adapter logic. + +### Latest supported adapters +- Qiskit Adapter (config value `mqss_qiskit`) +- Pennylane Adapter (config value `mqss_pennylane`) + +A complete list of configuration options will be listed and constantly updated for the upcoming releases + +## 🛠️ Upcoming Features +- Integration of the [Toolchain Project]([https://pages.github.com/](https://gitlab.lrz.de/qcbm/benchmark-comparison)) to the Framework for Simulator Benchmarks +- Improving the benchmark set for all types of benchmarks + +--- + +← [Back to Documentation Home](../index.md) \ No newline at end of file