engine: async NPU encoder compilation on init#733
Open
ncylich wants to merge 1 commit into
Open
Conversation
The NPU encoder mlpackages were compiled (compileModelAtURL) and loaded synchronously in Model::init, blocking startup for the entire CoreML compile, which is tens of seconds cold on the ANE path. Move the load onto a background thread kicked off early in init so init returns immediately; has_npu_*_encoder() and the source path gate on an atomic ready flag, so inference cleanly falls back to the CPU encoder until the NPU is compiled and loaded. The destructor joins the load before tearing down the encoders. Drop the init-time NPU vision soft-token override: the CPU vision encoder graph already sets the same image soft-token count, so the override is redundant and only existed to refine a value that matches. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
force-pushed
the
npu-async-compile-on-init
branch
from
June 16, 2026 22:29
e6bb211 to
c669281
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
NPU encoder mlpackages are compiled (
compileModelAtURL→.mlmodelc) and loaded synchronously insideModel::init. The CoreML compile is a one-time, per-device cost that is small on CPU/GPU but very large on the ANE path (tens of seconds cold; ~27s for the gemma-4-e2b encoders on an M-series Mac). Because it runs inline ininit, it blocks startup for the full compile on a fresh install.Change
Move the NPU encoder compile/load onto a background thread started early in
Model::init:start_npu_encoder_loads()launches the three encoder loads viastd::asyncand sets an atomicnpu_ready_flag when done;initreturns without waiting.has_npu_audio_encoder()/has_npu_vision_encoder()and the source-encode path gate onnpu_ready_, so any inference that happens before the NPU is ready cleanly falls back to the existing CPU encoder.~Modeljoins the load future before the encoders are destroyed.Trade-off
This does not make the compile cheaper — it defers it.
initreturns fast (~1s instead of ~27s cold); the compile runs in the background while early inferences use the CPU encoder, then the NPU takes over once ready. On warm launches (.mlmodelccached — the normal case) there is no compile and the NPU is used immediately.Validation
cactus-engine/tests/test_npu_startup.cpptimes init + two completes..mlmodelc, ANE forced):init~1.0s (was ~27s); first completes run on CPU fallback; total process wall time still includes the ~27s compile in the background..mlmodelccached):init~0.6–0.8s, completes uniform, NPU used; correct output, clean exit across repeated runs.