Fix llama-server and llama-bench - Stateless approach#13

Closed

wine99 wants to merge 171 commits intodev_backend_openvinofrom

stateless

Commits on Oct 15, 2025

Add ggml-openvino base files

YangleiZouIntel
authored and
wine99
committed
add openvino as optional backend for Llama.cpp ggml

zhanmyz
authored and
wine99
committed
* Configure the device(default CPU) that uses OpenVINO to compile the model

zhanmyz
authored and
wine99
committed
Solve the issue of abnormal model output caused by using OpenVINO ADD operator

zhanmyz
authored and
wine99
committed
Add OpenVINO MUL operator to GGML of Llama.cpp.

zhanmyz
authored and
wine99
committed
Add compile options

zhanmyz
authored and
wine99
committed
add OpenVINO frontend convert process steps

zhanmyz
authored and
wine99
committed
add get openvino available ops function

zhanmyz
authored and
wine99
committed
Add PoC of integration of openvino frontend. Main changes: ggml-ov-frontend-utils, GraphIterator, Decoder

yumengbo
authored and
wine99
committed
Implement GgmlOvDecoder. Add dump functions.

yumengbo
authored and
wine99
committed
Convert subgraph with add, sub, mul, div op to ov model and do infer on openvino device

yumengbo
authored and
wine99
committed
Add GGML_OV_FRONTEND option. Add readme.

yumengbo
authored and
wine99
committed
Change output for infer request to set output tensor. Support scale, view op.

yumengbo
authored and
wine99
committed
add GET_ROWS operator of OpenVINO to GGML of llama.cpp

zhanmyz
authored and
wine99
committed
Update build.md and add operation mapping(GGML to OpenVINO)

zhanmyz
authored and
wine99
committed
add the rms_norm operator implemented using OpenVINO to the GGML backend of llama.cpp

zhanmyz
authored and
wine99
committed
Fix issue for output memory copy of infer request

yumengbo
authored and
wine99
committed
Change to implementation following pytorch frontend

yumengbo
authored and
wine99
committed
Add support for UNARY SILU op . Fix pytorch impl bugs.

yumengbo
authored and
wine99
committed
Support Softmax op

yumengbo
authored and
wine99
committed
Support Softmax op

yumengbo
authored and
wine99
committed
Support ROPE op.

yumengbo
authored and
wine99
committed
Add support for RMS_NORM OP

zhanmyz
authored and
wine99
committed
Add MUL_MAT,CPY,CONT as operators implemented in OpenVINO for GGML backend

zhanmyz
authored and
wine99
committed
Move CPY from GGML OV Backend to OV Frontend

zhanmyz
authored and
wine99
committed
add implementation of MUL_MAT, CPY, CONT of GGML ops using OV ops

zhanmyz
authored and
wine99
committed
add implementation of CPY when the output tensor is non-contiguous

zhanmyz
authored and
wine99
committed
add tmp source code files

zhanmyz
authored and
wine99
committed
Execute singel CONT operator is OK

zhanmyz
authored and
wine99
committed
Execute CONT & VIEW operators in OV Frontend is OK

zhanmyz
authored and
wine99
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT graph conversion of consecutive OPs

zhanmyz
authored and
wine99
committed
OV Frontend supports GET_ROWS/RMS_NORM/MUL/MUL_MAT/ROPE/SCALE/SOFTMAX/ADD adjacent op graph conversion

zhanmyz
authored and
wine99
committed
Change the input parameter shape of CONT operator

zhanmyz
authored and
wine99
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
wine99
committed
Change the input and ouput node shape of MUL_MAT operator

zhanmyz
authored and
wine99
committed
change CONT and MULMAT input node shape

zhanmyz
authored and
wine99
committed
All adjacent ops can conversion but calculation result is wrong and need debugging

zhanmyz
authored and
wine99
committed
1. All operators implemented using OpenVINO can be successfully executed individually.

zhanmyz
authored and
wine99
committed
1. Update the implementation of CPY node when it's non-contiguous

zhanmyz
authored and
wine99
committed
Minor Update

zhanmyz
authored and
wine99
committed
Try to add VIEW node to OV Frontend and have some issues that need to be dealt with

zhanmyz
authored and
wine99
committed
1. In the Prompt process and predict first token stage, the PERMUTE node needs to be integrated into the OV Frontend

zhanmyz
authored and
wine99
committed
add debug info

zhanmyz
authored and
wine99
committed
Process Prompt and predict first token is OK

zhanmyz
authored and
wine99
committed
1. Solve the AC issue of Permute+VIEW and MULMAL issue in the phase of “1. Process Prompt and predict the first token”.

zhanmyz
authored and
wine99
committed
1. Delete some comments

zhanmyz
authored and
wine99
committed
* Use find_package in CMake to configure OpenVINO
wine99
committed
change op mappings to list in openvino_supports_op
wine99
committed
2nd+ token correct by fix CPY in OV, remove single op backend compute code
wine99
committed
Arbitrary token len (>32) work; Fix bug in mulmat
wine99
committed
FEAT: do PERMUTE eagerly
wine99
committed
FEAT: Add interleaved mode for ROPE
wine99
committed
REFACTOR: support weigts as constant
wine99
committed
STYLE: minor refactor
wine99
committed
PERF: share const nodes for weights for diff infer
wine99
committed
BUILD: update build doc, add cmake preset, add CACHE_DIR env var
wine99
committed
FEAT: improve debug capability
wine99
committed
PERF: compile once (dynamic graph + cache)
wine99
committed
Rebase - Bring up to date and fix build process

virajwad
authored and
wine99
committed
fix build error
wine99
committed
FIX: backend buffer type issue
wine99
committed
STYLE: clang-format
wine99
committed
FEAT: Add all conversion code from ov side
wine99
committed
PERF: favor low precision matmul
wine99
committed
STYLE and minor REFACTOR
wine99
committed
FIX: Re-add tensor names in cgraph, Add another case for RESHAPE
wine99
committed
FIX: input shape of KQ_mask
wine99
committed
PERF: add weight constant in parallel
wine99
committed
FIX: set_max_token_len
wine99
committed
PERF: use Slice+Concat in writing cache_v
wine99
committed
Update build doc
wine99
committed
Add cgraph tensor output name to OV op name
wine99
committed
Update openvino build instructions

ravi9
authored and
wine99
committed
Add initial NPU support
wine99
committed
draft NPU support version 2: prefill + kvcache
wine99
committed
NPU support version 2: prefill + kvcache
wine99
committed
Change due to ggml cgraph changes, not correct yet
wine99
committed
Change due to ggml cgraph changes, llama-3.2 CPU work
wine99
committed
Add AMD64 to CMakeLists
wine99
committed
Change due to ggml cgraph changes, all device work
wine99
committed
Refactor: clean, fix warning
wine99
committed
Update clang-format
wine99
committed
Statful transformation for CPU GPU
wine99
committed
Add SwiGLU
wine99
committed
Fuse to SDPA
wine99
committed
Replace Concat with Broadcast in MulMat for GQA
wine99
committed
Pull out indices creation for kv cache update
wine99
committed
Refactor: remove past_token_len from extra_inputs
wine99
committed
Fix Phi3 SwiGLU and SoftMax
wine99
committed
Pull out sin cos from rope
wine99
committed
Reduce memory: free ov weights node after graph conversion
wine99
committed
Fix CPY due to cgraph change
wine99
committed
Added OpenVINO CI/CD. Updated docs

ravi9
authored and
wine99
committed
Fix llama-cli
wine99
committed
Fix Phi3 ROPE; Add test-backend-ops
wine99
committed
Fix NPU
wine99
committed
Fix llama-bench; Clang-format
wine99
committed
Fix llama-perplexity
wine99
committed
temp. changes for mark decomp

cavusmustafa
authored and
wine99
committed
matmul in fp32
wine99
committed
mulmat input conversion fix

cavusmustafa
authored and
wine99
committed
mulmat type conversion update

cavusmustafa
authored and
wine99
committed
add mark decomp pass

cavusmustafa
authored and
wine99
committed
Revert changes in fuse_to_sdpa
wine99
committed
Update build.md

ravi9
authored and
wine99
committed
Fix test-backend-ops
wine99
committed
Skip test-thread-safety; Run ctest only in ci/run.sh
wine99
committed
Use CiD for NPU
wine99
committed
Optimize tensor conversion, improve TTFT
wine99
committed
Support op SET_ROWS
wine99
committed
Fix NPU
wine99
committed
Remove CPY
wine99
committed
Fix test-backend-ops
wine99
committed
Minor updates for raising PR
wine99
committed
Perf: RMS fused to OV internal RMS op
wine99
committed
Fix after rebasing
wine99
committed
Change openvino device_type to GPU; Enable flash_attn
wine99
committed
Update supports_buft and supports_op for quantized models
wine99
committed
Add quant weight conversion functions from genai gguf reader
wine99
committed
Quant models run with accuracy issue
wine99
committed
Fix accuracy: disable cpu_repack
wine99
committed
Fix CI; Disable test-backend-ops
wine99
committed
Fix Q4_1
wine99
committed
Fix test-thread-safety
wine99
committed
Fix test-backend-ops: Treat quantized tensors as weights
wine99
committed
Add NPU Q4_0 support
wine99
committed
NPU perf: eliminate zp
wine99
committed
Dequantize q4_1 q4_k q6_k for NPU
wine99
committed
Add custom quant type: q8_1_c, q4_0_128
wine99
committed
Set m_is_static=false as default in decoder
wine99
committed
Simpilfy translation of get_rows
wine99
committed
Fix after rebasing
wine99
committed
Improve debug util; Eliminate nop ReshapeReshape
wine99
committed
STYLE: make get_types_to_requant a function
wine99
committed
Support BF16 model
wine99
committed
Fix NPU compile
wine99
committed
WA for npu 1st token acc issue
wine99
committed
Apply EliminateZP only for npu
wine99
committed
Add GeGLU
wine99
committed
Fix Hunyuan
wine99
committed
Support iSWA
wine99
committed
Fix NPU accuracy
wine99
committed
Fix ROPE accuracy when freq_scale != 1
wine99
committed
Minor: not add attention_size_swa for non-swa model
wine99
committed
Minor refactor
wine99
committed
Add Q5_K to support phi-3-q4_k_m
wine99
committed
Requantize Q6_K (gs16) to gs32 on GPU
wine99
committed
Fix after rebasing
wine99
committed
Always apply Eliminate_ZP to fix GPU compile issue on some platforms
wine99
committed
kvcachefusion support

cavusmustafa
authored and
wine99
committed
env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added

cavusmustafa
authored and
wine99
committed
Fix for Phi3

cavusmustafa
authored and
wine99
committed
Fix llama-cli (need to run with --no-warmup)
wine99
committed
Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_size, iSWA model not working
wine99
committed
fix after rebasing
wine99
committed
Fix llama-3-8b and phi3-mini q4_0 NPU
wine99
committed
Update to OV-2025.3 and CMakeLists.txt

ravi9
authored and
wine99
committed
Add OV CI cache
wine99
committed
Apply CISC review and update CI to OV2025.3
ravi9
authored
Update CI to run OV dep install before build
ravi9
authored
Update OV dockerfile to use OV2025.3 and update build docs
ravi9
authored

Fix llama-server and llama-bench - Stateless approach#13wine99 wants to merge 171 commits intodev_backend_openvinoravi9/llama.cpp:dev_backend_openvinofrom statelessravi9/llama.cpp:statelessCopy head branch name to clipboard

Commits

Commits on Oct 15, 2025

Fix llama-server and llama-bench - Stateless approach#13
wine99 wants to merge 171 commits intodev_backend_openvinofrom
stateless