Hi,
I've been playing around with olive after being frustrated with the limited amounts of models available for my "AI" PC. Anyhow, I have been trying to run the following on a 32GB Lenovo Yoga Slim 7x as well as a dedicated cloud server with 32GB:
olive auto-opt \
--model_name_or_path "$CACHE_MODEL_PATH" \
$TRUST_FLAG \
--output_path "$OUTPUT_PATH" \
--device "$DEVICE" \
--provider "$PROVIDER" \
--use_ort_genai \
--precision "$PRECISION" \
--log_level "$LOG_LEVEL" \
$EXTRA_ARGSwith the following parameters
==========================================
QNN Model Conversion Container
==========================================
Model: Qwen/Qwen2.5-Coder-7B-Instruct
Output Directory: .
Output Name: qwen-coder-7b
Precision: int4
Device: npu
Provider: QNNExecutionProvider
Cache Directory: model-cache
==========================================
But no matter what I run into an OOM error:
[ 1299.228531] Out of memory: Killed process 11552 (olive) total-vm:43667312kB, anon-rss:31484340kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:64956kB oom_score_adj:0
Is this expected? If so is there any way to reduce the memory load while keeping the model "intelligent"? Originally I had planned on converting the devstral-23b from Mistral to finally run on my Qualcomm NPU, but as it seems that will continue to be a dream.
Hi,
I've been playing around with
oliveafter being frustrated with the limited amounts of models available for my "AI" PC. Anyhow, I have been trying to run the following on a 32GB Lenovo Yoga Slim 7x as well as a dedicated cloud server with 32GB:olive auto-opt \ --model_name_or_path "$CACHE_MODEL_PATH" \ $TRUST_FLAG \ --output_path "$OUTPUT_PATH" \ --device "$DEVICE" \ --provider "$PROVIDER" \ --use_ort_genai \ --precision "$PRECISION" \ --log_level "$LOG_LEVEL" \ $EXTRA_ARGSwith the following parameters
But no matter what I run into an OOM error:
Is this expected? If so is there any way to reduce the memory load while keeping the model "intelligent"? Originally I had planned on converting the devstral-23b from Mistral to finally run on my Qualcomm NPU, but as it seems that will continue to be a dream.