Sync with Microsoft ONNX Runtime - 03072026#1183
Closed
ai-fw-intg wants to merge 12 commits into
Closed
Conversation
… output indexing (microsoft#29264) ### Summary The CUDA `EmbedLayerNormalization` and `SkipLayerNormalization` kernels compute output write offsets (`row_index * hidden_size`) using 32-bit arithmetic. For very large output tensors the element count can exceed `INT32_MAX`, at which point the offset is no longer representable in 32 bits. Every output write index in these kernels is a pure function of the launch grid and `hidden_size` — there is no data-dependent write indexing — so the maximum index is exactly `output_element_count - 1`, which the host knows from the input shapes before launch. This PR adds a **host-side guard** in each op's `ComputeInternal` that computes the output element count in 64-bit arithmetic and returns a clear error when it exceeds the supported 32-bit indexing range. ### Design - **`EmbedLayerNormalization`** (`embed_layer_norm.cc`): `output_element_count = (int64)batch_size * sequence_length * hidden_size`, guarded with `ORT_RETURN_IF_NOT(... <= INT32_MAX, ...)`. - **`SkipLayerNormalization`** (`skip_layer_norm.cc`): `output_element_count = input->Shape().Size()` (output shares the input shape), same guard. - Kernels are **unchanged** — they keep the original int32 indexing, so there is no extra register/occupancy cost in the hot path. This is pure host-side validation. ### Behavior This **rejects** (rather than silently attempting) single-op LayerNorm outputs larger than 2³¹ elements — a regime no real BERT-family model produces (it would require a multi-GB single-op activation). For all supported shapes there is no behavior or numeric change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e inference (microsoft#29268) ### Description The `DecoderAttention` and `MultiHeadAttention` shape-inference functions guarded population of their optional `present_key` (output 1) and `present_value` (output 2) outputs with `getNumOutputs() > 1`, but then write output index 2. `present_key` and `present_value` are produced as a both-or-neither pair, so this requires all three outputs (`> 2`) to be present before populating them — matching the existing `BaseGroupQueryAttention` (`>= 3`) and `EmbedLayerNorm` guards. It also adds an output-index range check in `InferenceContextImpl::getOutputType` so an output index beyond the declared output count fails inference cleanly instead of indexing past the end of the outputs container, mirroring the existing `DataPropagationContextImpl::getOutputType` and `getInputType` behavior. ### Motivation and Context A model that declares fewer outputs than the optional present outputs could previously drive shape inference to access an output index that was not declared. This makes the guard consistent with the other attention-family contrib ops. ### Changes - `onnxruntime/core/graph/contrib_ops/bert_defs.cc` — require all present outputs before populating `present_key`/`present_value` in `DecoderAttention` and `MultiHeadAttention`. - `onnxruntime/core/graph/graph.cc` — add an output-index range check in `InferenceContextImpl::getOutputType`. - `onnxruntime/test/contrib_ops/attention_optional_outputs_shape_inference_test.cc` — regression tests covering omitted optional present outputs, the 3-output positive cases, and the MHA/DMMHA two-output cases. - Adds a contrib-op shape-inference output-index safety skill doc plus a one-line coding-convention note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…microsoft#29397) This pull request improves the handling of string tensors in the CPU implementation of the Loop operator, ensuring correct memory management and copy semantics for non-trivially-copyable types like `std::string`. It also adds comprehensive unit tests to verify these behaviors, especially for cases involving string scan outputs and loop-carried variables. **Enhancements for string tensor support:** * Updated `ConcatenateCpuOutput` in `loop.cc` to properly detect string tensors and use `std::copy` for concatenation, ensuring correct handling of heap-allocated string payloads and avoiding unsafe byte-wise copying. [[1]](diffhunk://#diff-2c8478657254a53c4ce09684960c925593395336e00c43ef672d9427722e3ff7R276-L282) [[2]](diffhunk://#diff-2c8478657254a53c4ce09684960c925593395336e00c43ef672d9427722e3ff7R289-R304) * Modified `OutputIterator::ZeroOutCurrent` in `scan_utils.h` to skip zeroing for string tensors, as their elements are already default-constructed and cannot be safely set with `memset`. **New and extended unit tests for string tensor scenarios:** * Added tests to `loop_test.cc` covering: - String scan outputs, including long strings that exceed the small-string-optimization threshold and multi-element outputs. - Loop-carried string variables to ensure they are copied correctly. - Zero-iteration cases to confirm empty string scan outputs are handled without errors.
…ssion initialization (microsoft#29250) This pull request introduces improved support and validation for external data in tensor attributes, particularly ensuring that external data in node attributes is properly validated and inlined, and that in-memory references are correctly rejected. Additionally, it introduces new tests to cover these scenarios, and refactors some utility functions in the test code for clarity and consistency. **External Data Handling and Validation:** * Updated `Graph::ConvertInitializersIntoOrtValues()` in `graph.cc` to: * Use `InlinedHashSet` for tracking validated external data paths for efficiency. * Add a new step that validates and inlines external data in node tensor attributes, ensuring that only file-based external data is accepted and in-memory references are rejected. This guarantees all execution providers have uniform access to attribute data. [[1]](diffhunk://#diff-e231a92b40d89409cc8e82436be0a15bc87ef95c93b303b9feaeab6e50c8835cL3842-R3842) [[2]](diffhunk://#diff-e231a92b40d89409cc8e82436be0a15bc87ef95c93b303b9feaeab6e50c8835cL3906-R3971) **Testing:** * Added comprehensive tests in `label_encoder_test.cc` to verify: * Valid external data in tensor attributes is loaded and inlined. * In-memory external data references in node attributes are rejected. * Duplicate key handling and singleton default tensor behavior. * Added a test in `tree_ensembler_test.cc` to ensure that in-memory external data references in node attributes are rejected, preventing invalid attribute usage. **Test Utility Refactoring:** * Refactored utility functions in `tree_ensembler_test.cc` and `treeregressor_test.cc`: * Renamed and standardized helper functions for array and string manipulations to improve code clarity (e.g., `MultiplyUpdateArray`, `MultiplyArraysValues`, `MultiplyUpdateArrayString`). [[1]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L47-R49) [[2]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L60-R62) [[3]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L90-R92) [[4]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L116-R127) [[5]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L174-R186) [[6]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L228-R230) [[7]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39L243-R245) [[8]](diffhunk://#diff-08b3495816c68f145657ecff63d7b5f3d56813586ec62f7324c22977e70e336bR6-R13) [[9]](diffhunk://#diff-08b3495816c68f145657ecff63d7b5f3d56813586ec62f7324c22977e70e336bL23-R25) **Test File Includes:** * Added necessary includes for file utilities and path handling in test files to support new test scenarios involving external data files. [[1]](diffhunk://#diff-0db6b4a4d9a180daab3cc2eab5da4437a2a7a3a2e81f53119f5e6298bd3dc4a5R7-R8) [[2]](diffhunk://#diff-9bce50df70fddc092ce6fc5351812c621343f682a15ca178c34e6dd9415e9a39R6-R7) [[3]](diffhunk://#diff-08b3495816c68f145657ecff63d7b5f3d56813586ec62f7324c22977e70e336bR6-R13)
…t_address (microsoft#29248) ### Description `vaip::process_ext_address` (in the VitisAI EP) decodes the in-memory "external data" address marker that ORT plants on initializers whose data lives in an in-process buffer. It only matched the little-endian tag `"*/_ORT_MEM_ADDR_/*"` (`kTensorProtoLittleEndianMemoryAddressTag`). This PR makes it also recognize the native-endian tag `"*/_ORT_NATIVE_ENDIAN_MEM_ADDR_/*"` (`kTensorProtoNativeEndianMemoryAddressTag`). ```cpp if (file == "*/_ORT_MEM_ADDR_/*" || file == "*/_ORT_NATIVE_ENDIAN_MEM_ADDR_/*") { auto addr = reinterpret_cast<const char*>(offset); return {addr, size}; } ``` ### Motivation and Context Two existing changes combined to break VitisAI model compilation: 1. microsoft#27404 split the single in-memory marker into a little-endian and a native-endian variant, and switched `TensorToTensorProto(use_tensor_buffer=true)` to emit the **native-endian** tag by default. Both `HasExternalDataInMemory` and the data readers treat the two tags equivalently. 2. microsoft#28709 added an explicit `ORT_ENFORCE(!utils::HasExternalDataInMemory(tensor), ...)` in `Graph::Graph`, rejecting any in-memory address marker found in a deserialized model protobuf. Because `process_ext_address` still only recognized the little-endian tag, in-memory initializers (e.g. quantized weights moved out of the `TensorProto` during graph optimization) are no longer detected in `vaip::model_clone`. They fall through to the verbatim-copy branch and the native-endian marker is carried into the cloned model proto. When ORT then constructs the cloned `Graph`, the new enforce fires and aborts compilation: ``` Initializer 'onnx::Conv_1575_quantized' references an ORT in-memory address marker, which is not allowed in a model protobuf. ``` This regression is observed on 1.27.0, where both changes are present (1.25.x / 1.26.x have the native-endian tag but not the enforce, so the marker leaked silently without crashing). Both tags encode the buffer address in the `offset` field; on little-endian hosts (the only platform this EP targets) the byte layout is identical, so they are decoded the same way. Recognizing both tags restores the previous behavior of converting these initializers into a lightweight reference inside `model_clone`, so the in-memory marker never reaches `Graph::Graph`.
## Summary Make cuDNN an optional runtime dependency for the CUDA Execution Provider and CUDA Plugin EP. The build still uses cuDNN headers, but provider binaries no longer directly depend on cuDNN shared libraries; cuDNN is loaded lazily when enabled and available, while no-cuDNN runs use native CUDA paths where available and report `NOT_IMPLEMENTED` for kernels that still require cuDNN. This also adds CI to validate no-cuDNN runtime environments. ## Key Changes | Area | Changes | |---|---| | CUDA EP runtime loading | Added a dynamic cuDNN loader and cuDNN symbol trampolines so CUDA provider binaries can avoid a direct cuDNN dependency. | | Provider options | Added `enable_cudnn`; removed `cudnn_path` from CUDA EP and CUDA Plugin EP provider configuration. | | CUDA Plugin EP | Wired optional cuDNN behavior through plugin EP config, kernel adapters, stream handles, and plugin utilities. | | Python preload behavior | Updated Python CUDA preload handling so cuDNN remains an optional dependency instead of an unconditional import/runtime requirement. | | Tests | Added/updated provider option coverage and CUDA Plugin EP no-cuDNN mode using `ORT_TEST_CUDA_PLUGIN_NO_CUDNN=1`. | | CI | Added Linux and Windows CUDA no-cuDNN workflows that build with cuDNN headers, exclude cuDNN from the runtime path, verify no direct cuDNN dependency, and run targeted tests. | | Documentation | Added `docs/CUDA_cuDNN_Optional_Design.md` and updated CUDA Plugin EP docs for no-cuDNN behavior and validation. | ## Testing Validated locally on Linux CUDA 13: - Rebuilt CUDA EP / CUDA Plugin EP with cuDNN headers available at build time. - Verified provider binaries have no direct cuDNN dependency: - `readelf -d ... | grep NEEDED | grep -i cudnn || echo "no cudnn DT_NEEDED"` - `ldd ... | grep -i cudnn || echo "no cudnn in ldd"` - Ran CUDA Plugin EP no-cuDNN validation: - `bash .env/cuda_130_plugin_no_cudnn.sh --test_plugin` - Result: `Ran 87 tests`, `OK (skipped=17)` Additional CI coverage is included for Linux and Windows no-cuDNN CUDA validation.
…osoft#28989) ### Description Builds out the standalone `model_package/` C library so a single library covers the full lifecycle of an ONNX Runtime model package: inspection, authoring, content-addressed shared assets, commit, prune, and validation. The library remains free of any dependency on ONNX Runtime itself. **Public C API (`model_package/include/model_package.h`)** - Lifecycle: `ModelPackage_Open` / `ModelPackage_New` / `ModelPackage_Close`, with `ModelPackageOpenOptions` controlling external-path access, symlink following, and strict unknown-field handling. - Inspection: a POD `ModelPackageInfo` tree (`ModelPackage_Info`) plus by-name lookups for components, variants, and per-namespace `executor_info` entries, and round-trip JSON getters that preserve fields unknown to the current build. - Path resolution: `ModelPackage_ResolveStringRef` implements the package's resolution rules — relative paths anchored at a base directory, `sha256:<hex>[/sub/path]` for shared-asset content, and portable vs installed confinement (absolute paths and `..` segments only allowed under `layout: "installed"`). - Shared assets: SHA-256 directory hashing, `ModelPackage_AddSharedAsset` (with reproducible-build URI check and an optional `copy_in` staging mode), `ModelPackage_RemoveSharedAsset`, and `ModelPackage_ResolveAssetUri`. Assets under `<package_root>/shared_assets/` are auto-discovered at `Open`. - Authoring: inline/external component setters, variant upsert/remove, per-namespace executor_info setters (inline and external), and package-level metadata / layout / `additional_metadata` setters. Mutations invalidate cached pointers in the mutated scope and its descendants. - Commit / prune / validate: `ModelPackage_Commit` writes the in-memory model to disk either in place or to a fresh `dest_root` ("save as"), with `PRESERVE` or `DENSE` write modes; `ModelPackage_Prune` reclaims unreferenced files under `shared_assets/` and tracked orphan variant/component directories left behind by removals; and `ModelPackage_Validate` runs schema, path-reachability, asset-rehash, and unknown-field checks and returns a JSON report. - Errors: opaque `ModelPackageStatus*` with a stable additive `ModelPackageErrorCode` enum (IO, schema, version, path confinement, asset missing, asset hash mismatch, not found, invalid arg, state). **Internal layout** The implementation is split into focused translation units: `manifest_parser`, `model_package_impl`, `authoring`, `commit_prune_validate`, `path_resolver`, `asset_hasher`, and an in-tree `sha256`. Shared error plumbing lives in `status_impl.h`. **ONNX Runtime integration (`onnxruntime/core/session/model_package/`)** The ORT-side glue is wired onto the library through the C inspection and path-resolution entry points. `model_package_context` now translates the library's info tree into ORT-internal structs and parses the `executor_info["ort"]` payload (`model_file`, `external_data`, `session_options`, `provider_options`). When a variant declares `external_data`, `CreateSessionForModelPackage` loads the model from a memory mapping and passes the resolved folder to the session via `kOrtSessionOptionsModelExternalInitializersFileFolderPath`, so external initializers — including those backed by a shared asset — are picked up at `Initialize` time. The experimental `OrtModelPackageApi_*_SinceV28` C entries introduced in microsoft#28990 are unchanged. **Documentation and tests** - `model_package/README.md` documents the on-disk layout, manifest and component schemas, shared-asset rules, path resolution, the authoring flow, and commit / prune / validate semantics. - `onnxruntime/core/session/model_package/README.md` documents the ORT consumer-side glue: the `executor_info["ort"]` schema, the variant selection algorithm, the session-creation contract, and the registered experimental C entries. - New library tests cover inspection, authoring, asset hashing, and commit (`model_package/tests/`). The ORT integration tests in `onnxruntime/test/autoep/test_model_package.cc` are reworked against the current C API surface. ### Motivation and Context ORT needs a single library that owns the model package format end to end — not just reading it, but producing it, validating it, and maintaining it on disk with content-addressed shared assets. Consolidating this behind one C API lets ORT, publisher tooling, and third-party consumers share the same parser, path-resolution rules, and on-disk invariants without each reimplementing them, and keeps the library independent of the ORT session runtime. --------- Co-authored-by: jambayk <jambayk@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: jambayk <jambay@github.com>
…cks (microsoft#29446) This pull request strengthens validation and error handling for the ConvTranspose operator in ONNX Runtime, particularly when using explicit `output_shape` attributes. It adds comprehensive checks to prevent inconsistent or invalid configurations, improves arithmetic safety to guard against integer overflows, and introduces a suite of targeted unit tests to verify these behaviors. **Validation and Error Handling Improvements:** * Added stricter input validation in `ConvTransposeAttributes::ComputePadAndOutputShape` to ensure that all relevant parameters (`output_shape`, input size, stride, kernel, dilation, and output padding) are within valid ranges and to provide clear error messages when they are not. This includes checks that all values are positive and that output padding is non-negative. * Added a consistency check to verify that the explicit `output_shape` is compatible with the input dimensions and convolution parameters, preventing buffer overruns and logical inconsistencies. **Arithmetic Safety:** * Updated `ComputeTotalPad` to use `SafeInt` for all intermediate arithmetic, ensuring that integer overflows are detected and handled safely instead of producing undefined behavior. **Testing Enhancements:** * Added a comprehensive set of unit tests for `ConvTranspose` with explicit `output_shape`, including cases for invalid, inconsistent, and overflow-prone configurations, as well as valid edge cases (e.g., 1D, 2D, and 3D, large batch sizes, group > 1, and cases requiring padding). These tests verify that invalid configurations are rejected and that valid ones work as expected. These changes collectively improve the robustness, correctness, and maintainability of the ConvTranspose operator's implementation and its handling of explicit output shapes.
…osoft#29450) Newer Homebrew versions refuse to load formulae from untrusted third-party taps. This adds `brew trust wix/brew` after tapping to allow the subsequent `brew install applesimutils` to succeed in React Native CI. **Changes:** - `.github/workflows/react_native.yml` - `tools/ci_build/github/azure-pipelines/templates/react-native-ci.yml` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description This PR introduces minor development patches as listed below. - Fix the OVIR EpCtx model export fixing the changes introduced by microsoft#28725 - Add test for OVIR EpCtx Export on sample model. - Update OV Toolkit version to 2026.2 - Add test for OVEP Workload Type Test. --------- Signed-off-by: Jonathan Clohessy <jonathan.clohessy@arm.com> Signed-off-by: bfilipek <bartlomiej.filipek@intel.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Christian Bourjau <christian.bourjau@quantco.com> Co-authored-by: Jaswanth Gannamaneni <jaswanth.gannamaneni@intel.com> Co-authored-by: Ryan Metcalfe <107415876+RyanMetcalfeInt8@users.noreply.github.com> Co-authored-by: jatinwadhwa921 <110383850+jatinwadhwa921@users.noreply.github.com> Co-authored-by: derdeljan-msft <derdeljan@microsoft.com> Co-authored-by: Jonathan Clohessy <jonathan.clohessy@arm.com> Co-authored-by: Akshay Sonawane <111780983+apsonawane@users.noreply.github.com> Co-authored-by: Christopher Warrington <chwarr@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ishwar Raut <iraut@nvidia.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Xinpeng Dou <15529241576@163.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: adrastogi <aditya.rastogi@microsoft.com> Co-authored-by: Aditya Rastogi <adityar@ntdev.microsoft.com> Co-authored-by: qti-hungjuiw <hungjuiw@qti.qualcomm.com> Co-authored-by: qti-yuduo <yuduow@qti.qualcomm.com> Co-authored-by: Pradeep Sakhamoori <psakhamoori@microsoft.com> Co-authored-by: Preetha Veeramalai <preetha.veeramalai@intel.com> Co-authored-by: Klimenko, Mikhail <mikhail.klimenko@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel.com> Co-authored-by: Garth Long <garth.long@intel.com> Co-authored-by: MayureshV1 <47039074+MayureshV1@users.noreply.github.com> Co-authored-by: Eric Crawford <eric.r.crawford@intel.com> Co-authored-by: Vishnudas Thaniel S <vishnudas.thaniel.s@intel.com> Co-authored-by: Javier Martinez <javier.e.martinez@intel.com> Co-authored-by: Adam Pocock <adam.pocock@oracle.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: mingyue <131847423+mingyueliuh@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Susanta Bhattacharjee <susanta.bhattacharjee@intel.com> Co-authored-by: jatinwadhwa921 <jatin.wadhwa@intel.com> Co-authored-by: Jozef Wludzik <jozef.wludzik@intel.com> Co-authored-by: Bartlomiej Filipek <bartlomiej.filipek@intel.com> Co-authored-by: Kotomi-Du <yaru.du@intel.com> Co-authored-by: Rajeev Sekar <rajeevsekar21@gmail.com> Co-authored-by: liang <gxgaoliang@126.com> Co-authored-by: n1harika <niharika.sathish@intel.com> Co-authored-by: Mayuresh M Varerkar <mayuresh.m.varerkar@intel.com> Co-authored-by: Mikhail Dvoretckii <mikhail.dvoretckii@intel.com> Co-authored-by: bopeng1234 <bo.peng@intel.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: fs-eire <7679871+fs-eire@users.noreply.github.com> Co-authored-by: Wenqin Yang <wenqin.yang@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: xieofxie <xieofxie@126.com> Co-authored-by: hualxie <hualxie@microsoft.com> Co-authored-by: Jiajia Qin <jiajiaqin@microsoft.com> Co-authored-by: Joshua Lochner <admin@xenova.com> Co-authored-by: Christian Bourjau <cbourjau@users.noreply.github.com> Co-authored-by: Xiaofei Han <xiaofeihan@microsoft.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: chunghow-qti <chunghow@qti.qualcomm.com> Co-authored-by: Guenther Schmuelling <guschmue@microsoft.com> Co-authored-by: Jiawei Shao <jiawei.shao@intel.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: czekun <chen.zekun@intel.com> Co-authored-by: Ryan Metcalfe <ryan.metcalfe@intel.com> Co-authored-by: Jaskaran Singh Nagi <jaskaran.singh.nagi@intel.com> Co-authored-by: ai-fw-intg <sys_ai_fw_intg@intel.com> Co-authored-by: Rajeev Sekar <rajeev.sekar@intel.com> Co-authored-by: RajeevSekar <117911837+RajeevSekar@users.noreply.github.com> Co-authored-by: Nazanin Beheshti <nazanin.beheshti@intel.com>
…29451) ### Description Add a batched GEMV for the small-M range (M = 2..16) of CUDA `MatMulNBits` (4-bit and 8-bit), using the standard `[N, blocks, blob]` weight layout with **no prepacking**. Previously, `MatMulNBits` had a fast single-row GEMV for M = 1 (decode), but for M > 1 it fell back to full weight dequantization + cuBLAS, which dequantizes the entire weight matrix regardless of M. That fallback is therefore flat across small M and dominates latency at the row counts that matter for multi-row decode. The new half/bf16 path uses a `CtaM x CtaN` register-tiled kernel that streams the quantized weight once per block row and reuses each activation load across columns, so latency scales with M. M = 1 decode is unchanged. No prepacking is used, so there is no extra resident weight memory and no GEMM tactic profiling at session init. Also adds `profile_matmul_nbits.py` (same style as `profile_qmoe_gemv.py`, parseable with `parse_nsys.py`) and a docs experiment log. **Before (main: M > 1 dequant + cuBLAS) vs After (batched small-M GEMV)** — A100, block_size 32, fp16, average op latency in microseconds (lower is better). #### 4-bit (M = 2..16) Before: | matrix | K | N | M=1 | M=2 | M=4 | M=8 | M=16 | |---------|-------|--------|-------|--------|--------|--------|--------| | qkv | 4096 | 4096 | 25.9 | 77.2 | 69.3 | 69.5 | 69.9 | | o_proj | 4096 | 4096 | 23.2 | 69.2 | 69.1 | 69.3 | 69.7 | | gate_up | 4096 | 12288 | 39.3 | 172.0 | 172.1 | 172.1 | 172.4 | | down | 12288 | 4096 | 38.8 | 174.8 | 175.1 | 175.4 | 175.6 | | lm_head | 4096 | 151936 | 301.6 | 1868.0 | 1871.1 | 1877.8 | 1885.1 | After: | matrix | K | N | M=1 | M=2 | M=4 | M=8 | M=16 | |---------|-------|--------|-------|-------|-------|-------|--------| | qkv | 4096 | 4096 | 25.4 | 30.1 | 32.3 | 43.2 | 70.0 | | o_proj | 4096 | 4096 | 22.6 | 26.4 | 28.1 | 36.7 | 58.7 | | gate_up | 4096 | 12288 | 37.5 | 43.5 | 49.5 | 71.9 | 125.9 | | down | 12288 | 4096 | 37.9 | 47.4 | 52.9 | 76.8 | 150.1 | | lm_head | 4096 | 151936 | 300.7 | 329.9 | 424.1 | 635.3 | 1226.9 | Speedup (before / after): | matrix | M=2 | M=4 | M=8 | M=16 | |---------|-------|-------|-------|-------| | qkv | 2.56x | 2.15x | 1.61x | 1.00x | | o_proj | 2.62x | 2.46x | 1.89x | 1.19x | | gate_up | 3.95x | 3.48x | 2.39x | 1.37x | | down | 3.69x | 3.31x | 2.28x | 1.17x | | lm_head | 5.66x | 4.41x | 2.96x | 1.54x | #### 8-bit (M = 2..5) 8-bit weights are twice the bytes of 4-bit and the GEMV runs on CUDA cores, so it crosses over to the dequantize + cuBLAS (tensor-core) fallback at a lower M; the batched path covers M = 2..5 and M >= 6 keeps the fallback. Before: | matrix | K | N | M=1 | M=2 | M=3 | M=4 | M=5 | |---------|-------|--------|-------|--------|--------|--------|--------| | qkv | 4096 | 4096 | 36.2 | 80.6 | 72.9 | 72.8 | 73.3 | | o_proj | 4096 | 4096 | 31.1 | 73.2 | 72.7 | 73.6 | 73.4 | | gate_up | 4096 | 12288 | 63.5 | 184.4 | 184.2 | 184.1 | 184.3 | | down | 12288 | 4096 | 67.8 | 187.3 | 187.4 | 187.8 | 188.0 | | lm_head | 4096 | 151936 | 535.9 | 2025.0 | 2025.9 | 2028.1 | 2029.8 | After: | matrix | K | N | M=1 | M=2 | M=3 | M=4 | M=5 | |---------|-------|--------|-------|-------|-------|--------|--------| | qkv | 4096 | 4096 | 36.2 | 46.1 | 48.7 | 57.9 | 67.0 | | o_proj | 4096 | 4096 | 31.6 | 39.5 | 48.6 | 57.9 | 67.1 | | gate_up | 4096 | 12288 | 63.4 | 80.9 | 104.6 | 128.9 | 152.6 | | down | 12288 | 4096 | 68.0 | 96.2 | 119.3 | 146.4 | 172.0 | | lm_head | 4096 | 151936 | 536.3 | 647.0 | 896.9 | 1157.1 | 1420.1 | Speedup (before / after): | matrix | M=2 | M=3 | M=4 | M=5 | |---------|-------|-------|-------|-------| | qkv | 1.75x | 1.50x | 1.26x | 1.09x | | o_proj | 1.85x | 1.50x | 1.27x | 1.09x | | gate_up | 2.28x | 1.76x | 1.43x | 1.21x | | down | 1.95x | 1.57x | 1.28x | 1.09x | | lm_head | 3.13x | 2.26x | 1.75x | 1.43x | ### Motivation and Context The small-M (M = 2..16) regime is exactly what speculative decoding hits when verifying a block of draft tokens: each step runs the target model on a handful of rows rather than a single token. With the previous dispatch, that step paid the full-dequant + cuBLAS cost (flat ~172 us for an MLP projection, ~1.9 ms for the lm_head even at M = 2), which erased much of the speculative-decoding speedup over greedy decoding. Routing these row counts through a batched GEMV that scales with M (2.6-5.7x faster at M = 2, at or above parity through M = 16) restores the benefit, without the resident-memory and session-init costs of a prepacked weight layout. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.