Skip to content

engine: async NPU encoder compilation on init#733

Open
ncylich wants to merge 1 commit into
mainfrom
npu-async-compile-on-init
Open

engine: async NPU encoder compilation on init#733
ncylich wants to merge 1 commit into
mainfrom
npu-async-compile-on-init

Conversation

@ncylich

@ncylich ncylich commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

NPU encoder mlpackages are compiled (compileModelAtURL.mlmodelc) and loaded synchronously inside Model::init. The CoreML compile is a one-time, per-device cost that is small on CPU/GPU but very large on the ANE path (tens of seconds cold; ~27s for the gemma-4-e2b encoders on an M-series Mac). Because it runs inline in init, it blocks startup for the full compile on a fresh install.

Change

Move the NPU encoder compile/load onto a background thread started early in Model::init:

  • start_npu_encoder_loads() launches the three encoder loads via std::async and sets an atomic npu_ready_ flag when done; init returns without waiting.
  • has_npu_audio_encoder() / has_npu_vision_encoder() and the source-encode path gate on npu_ready_, so any inference that happens before the NPU is ready cleanly falls back to the existing CPU encoder.
  • ~Model joins the load future before the encoders are destroyed.
  • Removed the init-time NPU vision soft-token override: the CPU vision-encoder graph already sets the same image soft-token count (verified equal), so it was redundant.

Trade-off

This does not make the compile cheaper — it defers it. init returns fast (~1s instead of ~27s cold); the compile runs in the background while early inferences use the CPU encoder, then the NPU takes over once ready. On warm launches (.mlmodelc cached — the normal case) there is no compile and the NPU is used immediately.

Validation

cactus-engine/tests/test_npu_startup.cpp times init + two completes.

  • Cold (no .mlmodelc, ANE forced): init ~1.0s (was ~27s); first completes run on CPU fallback; total process wall time still includes the ~27s compile in the background.
  • Warm (.mlmodelc cached): init ~0.6–0.8s, completes uniform, NPU used; correct output, clean exit across repeated runs.

The NPU encoder mlpackages were compiled (compileModelAtURL) and loaded
synchronously in Model::init, blocking startup for the entire CoreML
compile, which is tens of seconds cold on the ANE path. Move the load
onto a background thread kicked off early in init so init returns
immediately; has_npu_*_encoder() and the source path gate on an atomic
ready flag, so inference cleanly falls back to the CPU encoder until the
NPU is compiled and loaded. The destructor joins the load before tearing
down the encoders.

Drop the init-time NPU vision soft-token override: the CPU vision
encoder graph already sets the same image soft-token count, so the
override is redundant and only existed to refine a value that matches.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the npu-async-compile-on-init branch from e6bb211 to c669281 Compare June 16, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant