feat: Hybrid unified/recurrent cache#13276

Closed

gabe-l-hart wants to merge 21 commits intoggml-org:gg/kv-cache-simplify-part3from

gabe-l-hart:HybridCache

Commits on May 27, 2025

tests: Initial unit tests for memory hierarchy
gabe-l-hart
committed
build: Add build step for test-memory on non-windows builds
gabe-l-hart
committed
fix(tests): Fix constructors in tests for signature changes after rebase
gabe-l-hart
committed
tests(wip): More robust test for unified cache
gabe-l-hart
committed
feat: Add can_seq_rm API to llama_kv_cache API
gabe-l-hart
committed
feat: Move layer_filter_cb up to llama_kv_cache
gabe-l-hart
committed
feat: Add layer filter to recurrent cache
gabe-l-hart
committed
feat: Initial implementation of llama_kv_cache_hybrid
gabe-l-hart
committed
feat: Add llama_model_is_hybrid API call
gabe-l-hart
committed
feat: Add c++ side constants for attention layer indices hparam
gabe-l-hart
committed
feat: Add support for distinguishing recurrent vs non-recurrent layers in hparams
gabe-l-hart
committed
feat: Auto-fill hparams.recurrent_layer_arr based on whether the model is recurrent
gabe-l-hart
committed
feat: Instantiate hybrid cache for hybrid models (currently none)
gabe-l-hart
committed
refactor: rename *_is_hybrid -> *_is_hybrid_recurrent
gabe-l-hart
committed
fix: Fix indexing into k_l for recurrent cache with filter
gabe-l-hart
committed
fix: Use per-layer sizing everywhere in kv caches
gabe-l-hart
committed
fix: Remove unused kv cache methods after rebase
gabe-l-hart
committed
fix(tests): Fix constructors in tests for signature changes after rebase
gabe-l-hart
committed
feat: Add split_equal to init(...) signature
gabe-l-hart
committed
fix: Overhaul hybrid cache for refactor part3 (::init interface)
gabe-l-hart
committed
tests(wip): Comment out broken test for now and fix other constructor signatures
gabe-l-hart
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Hybrid unified/recurrent cache#13276

feat: Hybrid unified/recurrent cache#13276
gabe-l-hart wants to merge 21 commits intoggml-org:gg/kv-cache-simplify-part3from
gabe-l-hart:HybridCache

Commits on May 27, 2025

tests: Initial unit tests for memory hierarchy

build: Add build step for test-memory on non-windows builds

fix(tests): Fix constructors in tests for signature changes after rebase

tests(wip): More robust test for unified cache

feat: Add can_seq_rm API to llama_kv_cache API

feat: Move layer_filter_cb up to llama_kv_cache

feat: Add layer filter to recurrent cache

feat: Initial implementation of llama_kv_cache_hybrid

feat: Add llama_model_is_hybrid API call

feat: Add c++ side constants for attention layer indices hparam

feat: Add support for distinguishing recurrent vs non-recurrent layers in hparams

feat: Auto-fill hparams.recurrent_layer_arr based on whether the model is recurrent

feat: Instantiate hybrid cache for hybrid models (currently none)

refactor: rename _is_hybrid -> _is_hybrid_recurrent

fix: Fix indexing into k_l for recurrent cache with filter

fix: Use per-layer sizing everywhere in kv caches

fix: Remove unused kv cache methods after rebase

fix(tests): Fix constructors in tests for signature changes after rebase

feat: Add split_equal to init(...) signature

fix: Overhaul hybrid cache for refactor part3 (::init interface)

tests(wip): Comment out broken test for now and fix other constructor signatures

feat: Hybrid unified/recurrent cache#13276gabe-l-hart wants to merge 21 commits intoggml-org:gg/kv-cache-simplify-part3ggml-org/llama.cpp:gg/kv-cache-simplify-part3from gabe-l-hart:HybridCachegabe-l-hart/llama.cpp:HybridCacheCopy head branch name to clipboard

Commits

Commits on May 27, 2025

feat: Hybrid unified/recurrent cache#13276
gabe-l-hart wants to merge 21 commits intoggml-org:gg/kv-cache-simplify-part3from
gabe-l-hart:HybridCache