Possible solution for GGML_NUMA_MIRROR

[wkgcass]

TLDR: Replicate models on each NUMA. On my platform, pure CPU inference of QwQ-32B FP16 improved from ~6.6 token/s to ~10.7 token/s, and DeepSeek R1 671B Q8 from ~7.2 token/s to ~9.7 token/s. You can find the modified llama.cpp version here.

---

On a dual socket system, cross-NUMA access is extremely slow.
The most memory-bandwidth-consuming component during LLM inference execution is the model itself's memory usage.

To maximize utilization of the multi-CPU-platform’s total memory bandwidth:
We can replicate one copy of our neural network per available numa node
Then use local copies when doing calculations (i.e., always work with your current NUMA node's own replica)

This would theoretically achieve double bandwidth than a single numa.

Model addresses are primarily stored in tensor->data. To enable access via different NUMAs' data, we modify:

tensor->__data[2](assuming two NUMAs) instead of single pointer.

When setting values for tensor->data:
Assign each NUMA its respective memory region address

How to know the per-NUMA memory locations?
Leverage Linux mmap’s capability allowing specifying virtual addresses.
During mapping phase allocate specific regions for model copies across numa nodes
Example:
NUMA node 0 uses base address starting at 0x200000000000
NUMA node 1: 0x400000000000

Assignment logic would be：
If a given data-pointer falls within [0x2... ~0x4...] range,
then __data[0] retains original value, while
__data[1]=original_data + (offset between the two NUMAs' bases)

When accessing tensor->data during runtime，
the thread's current NUMA ID is retrieved via its TLS storage.
We should also bind threads to specific cores/numa nodes.

To implement this:
Modify all instances of tensor->data accesses in codebase
Create helper functions：

• tensor_data() returns appropriate address for local numa node.
• tensor_set_data() populates both copies when setting values

This requires changing about 700 lines of code.

---

Testing Platform: 9275f × 2 + DDR5-6000MT/s×(2x12 channels)
Model Used:QwQ-32B FP16
Codebase Commit:1e2f78a00450593e2dfa458796fcdd9987300dfc

Test Scenarios:
Scenario 1 - Single NUMA Mode：
BIOS configures all memory into single unified node with data spread between both physical nodes

• QwQ-32B FP16 generate = 1085, speed = 6.66, power = 798W
• DeepSeek R1 671B Q8 generate = 1022, speed = 7.19, power = 753W

Scenario 2 - Two NUMA nodes with numactl --interleave=0,1

• QwQ-32B generate = 1399, speed = 6.82, power = 806W
• DeepSeek R1 671B Q8 generate = 1056, speed = 7.23, power = 728W

Scenario 3 - New GGML_NUMA_MIRROR scheme proposed above

• QwQ-32B generate = 1344, speed = 10.80, power = 884W
• DeepSeek R1 671B Q8 generate = 1084, speed = 9.67, power = 762W

* power consumption includes the whole server case and one not-in-use GPU.

---

Here's the code for you to try out: vproxy-tools/llama.cpp

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_NUMA_MIRROR=ON

The tensor->data modifications are stored here

[ubergarm]

Hey thanks for sharing this implementation. I'd been wondering if it was possible to do data parallel onto each NUMA node myself. ktransformers supports this too.

I updated my discussion on NUMA performance to link this, and hopefully I can try it out at some point. Cheers!

EDIT Fixed Link copy paste error! thx!

[jukofyork]

This looks really interesting!

@ubergarm I'm getting a 404 for the link?

[usrlocalben]

CPU inference of QwQ-32B FP16 improved from ~6.6 token/s to ~10.7 token/s, and DeepSeek R1 671B Q8 from ~7.2 token/s to ~9.7 token/s.

Can you give any detail on your measurement method? I assume this is a very small context.
If there is some reason I should be getting e.g. >7 tok/sec with vanilla on a 2S Epyc machine, I'd love to know how. (And I don't think 4800 vs 6000 accounts for it either)

Regardless, I tried out the mirror.

2S Epyc 9115, each w/12x (i.e. 24x) DDR5-4800, total 1.5TB
NPS1 for vproxy-tools, NPS0 for vanilla and ik
Kernel runs with mitigations=off
The prompt is 3000 input tokens of T&C boilerplate e.g. Summarize this agreement, and generates about 1000 tokens.
Figures are in millis/tok as comparing tok/sec is not as useful.
I didn't bother to take a bunch of samples, just N=1 for each, on a quiet system with the same prompt.

It seems clear that porting the mirror impl. to the ik fork should make the best available version. Otherwise, it's not an easy choice to make between ik and the mirror since it will depend on the use-case which is better served. Reconfiguring e.g. NPS0 vs. NPS1 is a painful 15 minute process on my machine, since it requires two board resets, at least one with full duration POST.
edit: Running the ik fork with -mla 3 -fa -fmoe as suggested by @ikawrakow below improved results substantially

It's unfortunate that the full context won't fit w/Q8 on a 24x 64GB machine though, only about 1/4 of that I think.

DeepSeek-R1 Q8

build	prompt (ms/t)	t/s	pct	gen (ms/t)	t/s	pct
vanilla	64	15.6	1.00	308	3.2	1.00
vproxy-tools	47	21.3	0.73	259	3.9	0.84
ik	32	31.2	0.50	194	5.2	0.63

DeepSeek-R1 Q4

build	prompt (ms/t)	t/s	pct	gen (ms/t)	t/s	pct
vanilla	54	18.5	1.00	258	3.9	1.00
vproxy-tools	47	21.3	0.87	208	4.8	0.81
ik	24	41.0	0.44	132	7.6	0.51

A few notes on hugepages etc. - my first time using them.

• 2MB pages on 2S with the proc/sys entry implicitly dividing by 2, with a goal of loading 2 copies makes for clunky calculation
• user must know the in-ram size of model ahead of time, in terms of 2MB pages
• exe doesn't tell us what it's about to allocate, it just fails--perhaps after a long load time--e.g. if one used *1000 instead of *1024: no failure until the last chunk
• exe doesn't clean up hugepages at termination. very confusing as the next run will reuse them and happily give completely erroneous inference output
• requires reconfiguring kernel/hugepages to change models (not that big a deal as model is usually static on this kind of setup)
• mentions of mmap in the log now bring some ambiguity vs. normal llamacpp since mmap now refers to hugepages and not GGUF files

Probably a naive thought, but if one was going to spend 2x RAM on perf, wouldn't there be other things to explore such as storing materialized transposed matrices so dot-products would be faster wrt. cacheline hit and prefetch? 🤔

[wkgcass]

Hi,

Can you give any detail on your measurement method? I assume this is a very small context.

Yes the context is small, with only a few input tokens and about 1000~1300 generated tokens as shown above.

---

exe doesn't clean up hugepages at termination.

This is by design. We can skip the model loading process between program rebooting.
This helps when debugging or trying out different parameters.

And we could easily clean the hugepages by running rm /dev/hugepages/llama-node* when we want to switch to a different model.

---

And I don't think 4800 vs 6000 accounts for it either

Sure, I also believe that the memory frequency comparing 4800 and 6000 won't matter that much.
We can test the memory throughput to know for sure.

Here's my output:

numactl -N ? --interleave=? mbw -n 10 -t0 1024

wkgcass@exp0:~$ numactl -N 0 --interleave=0 mbw -n 10 -t0 1024
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Getting down to business... Doing 10 runs per test.
0	Method: MEMCPY	Elapsed: 0.04806	MiB: 1024.00000	Copy: 21308.030 MiB/s
1	Method: MEMCPY	Elapsed: 0.04808	MiB: 1024.00000	Copy: 21297.837 MiB/s
2	Method: MEMCPY	Elapsed: 0.04809	MiB: 1024.00000	Copy: 21293.408 MiB/s
3	Method: MEMCPY	Elapsed: 0.04809	MiB: 1024.00000	Copy: 21294.294 MiB/s
4	Method: MEMCPY	Elapsed: 0.04807	MiB: 1024.00000	Copy: 21302.711 MiB/s
5	Method: MEMCPY	Elapsed: 0.04810	MiB: 1024.00000	Copy: 21287.211 MiB/s
6	Method: MEMCPY	Elapsed: 0.04812	MiB: 1024.00000	Copy: 21280.575 MiB/s
7	Method: MEMCPY	Elapsed: 0.04810	MiB: 1024.00000	Copy: 21287.654 MiB/s
8	Method: MEMCPY	Elapsed: 0.04811	MiB: 1024.00000	Copy: 21283.229 MiB/s
9	Method: MEMCPY	Elapsed: 0.04810	MiB: 1024.00000	Copy: 21287.654 MiB/s
AVG	Method: MEMCPY	Elapsed: 0.04809	MiB: 1024.00000	Copy: 21292.257 MiB/s
wkgcass@exp0:~$ numactl -N 0 --interleave=1 mbw -n 10 -t0 1024
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Getting down to business... Doing 10 runs per test.
0	Method: MEMCPY	Elapsed: 0.06901	MiB: 1024.00000	Copy: 14838.859 MiB/s
1	Method: MEMCPY	Elapsed: 0.06904	MiB: 1024.00000	Copy: 14831.981 MiB/s
2	Method: MEMCPY	Elapsed: 0.06907	MiB: 1024.00000	Copy: 14825.539 MiB/s
3	Method: MEMCPY	Elapsed: 0.06905	MiB: 1024.00000	Copy: 14830.693 MiB/s
4	Method: MEMCPY	Elapsed: 0.06905	MiB: 1024.00000	Copy: 14828.974 MiB/s
5	Method: MEMCPY	Elapsed: 0.06904	MiB: 1024.00000	Copy: 14831.767 MiB/s
6	Method: MEMCPY	Elapsed: 0.06906	MiB: 1024.00000	Copy: 14828.330 MiB/s
7	Method: MEMCPY	Elapsed: 0.06905	MiB: 1024.00000	Copy: 14829.619 MiB/s
8	Method: MEMCPY	Elapsed: 0.06905	MiB: 1024.00000	Copy: 14830.048 MiB/s
9	Method: MEMCPY	Elapsed: 0.06906	MiB: 1024.00000	Copy: 14827.471 MiB/s
AVG	Method: MEMCPY	Elapsed: 0.06905	MiB: 1024.00000	Copy: 14830.327 MiB/s
wkgcass@exp0:~$ numactl -N 1 --interleave=1 mbw -n 10 -t0 1024
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Getting down to business... Doing 10 runs per test.
0	Method: MEMCPY	Elapsed: 0.04220	MiB: 1024.00000	Copy: 24262.528 MiB/s
1	Method: MEMCPY	Elapsed: 0.04217	MiB: 1024.00000	Copy: 24285.545 MiB/s
2	Method: MEMCPY	Elapsed: 0.04216	MiB: 1024.00000	Copy: 24287.273 MiB/s
3	Method: MEMCPY	Elapsed: 0.04214	MiB: 1024.00000	Copy: 24297.646 MiB/s
4	Method: MEMCPY	Elapsed: 0.04214	MiB: 1024.00000	Copy: 24298.799 MiB/s
5	Method: MEMCPY	Elapsed: 0.04207	MiB: 1024.00000	Copy: 24341.542 MiB/s
6	Method: MEMCPY	Elapsed: 0.04210	MiB: 1024.00000	Copy: 24322.463 MiB/s
7	Method: MEMCPY	Elapsed: 0.04209	MiB: 1024.00000	Copy: 24327.085 MiB/s
8	Method: MEMCPY	Elapsed: 0.04209	MiB: 1024.00000	Copy: 24329.975 MiB/s
9	Method: MEMCPY	Elapsed: 0.04213	MiB: 1024.00000	Copy: 24306.874 MiB/s
AVG	Method: MEMCPY	Elapsed: 0.04213	MiB: 1024.00000	Copy: 24305.951 MiB/s
wkgcass@exp0:~$ numactl -N 1 --interleave=0 mbw -n 10 -t0 1024
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Getting down to business... Doing 10 runs per test.
0	Method: MEMCPY	Elapsed: 0.06179	MiB: 1024.00000	Copy: 16572.261 MiB/s
1	Method: MEMCPY	Elapsed: 0.06174	MiB: 1024.00000	Copy: 16585.413 MiB/s
2	Method: MEMCPY	Elapsed: 0.06176	MiB: 1024.00000	Copy: 16580.042 MiB/s
3	Method: MEMCPY	Elapsed: 0.06173	MiB: 1024.00000	Copy: 16588.100 MiB/s
4	Method: MEMCPY	Elapsed: 0.06176	MiB: 1024.00000	Copy: 16579.506 MiB/s
5	Method: MEMCPY	Elapsed: 0.06177	MiB: 1024.00000	Copy: 16577.358 MiB/s
6	Method: MEMCPY	Elapsed: 0.06186	MiB: 1024.00000	Copy: 16553.776 MiB/s
7	Method: MEMCPY	Elapsed: 0.06192	MiB: 1024.00000	Copy: 16536.934 MiB/s
8	Method: MEMCPY	Elapsed: 0.06190	MiB: 1024.00000	Copy: 16542.277 MiB/s
9	Method: MEMCPY	Elapsed: 0.06194	MiB: 1024.00000	Copy: 16532.395 MiB/s
AVG	Method: MEMCPY	Elapsed: 0.06182	MiB: 1024.00000	Copy: 16564.781 MiB/s

---

We could also consider the possibility that the performance degradation comes from the cpu, since a 16 core 2.6/3.3GHz cpu could be 'not fast enough'. That might be why ik performs much better when generating since it's bundled with lots of cpu optimizations. (Normally the generation speed shouldn't be affected too much since it should be memory bounded)

[ubergarm]

Thanks for the updates @usrlocalben , inspired me to give it a try too on a dual socket Intel Xeon 6980P rig with 1.5TB RAM configured in BIOS to SNC=Disabled which is roughly equivalent to your dual socket AMD Epyc in NPS1. No GPU on this rig, so testing CPU only.

First off, it was a pain to compile on intel given backend amx code did not contain the updated dual tensor stuff. So i just ripped out the amx stuff for now for this test.

Eventually got it to actually run and does seem to use the Explicit Huge Pages which I allocated. It takes a looong time to start up haha...

When accessing tensor->data during runtime，
the thread's current NUMA ID is retrieved via its TLS storage.
We should also bind threads to specific cores/numa nodes.

To implement this:
Modify all instances of tensor->data accesses in codebase
Create helper functions：

tensor_data() returns appropriate address for local numa node.
tensor_set_data() populates both copies when setting values

This requires changing about 700 lines of code.

Is this something I'm supposed to do, or is already in the codebase? I ask because it seems like all 64 threads running were coming from the first NUMA node and not distributed between the two (i would expect 32 threads per NUMA node.)

Anyway, here are my logs if anyone else wants to have a stab at it. The idea is promising as ktransformers has shown some success, and doing per numa node "data parallel" allocation seems like it is useful in other inference engines as well

So unless I figure out how to spread the allocated threads between the two NUMA nodes, not worth benchmarking yet imo.

Finally, like @usrlocalben says:

It seems clear that porting the mirror impl. to the ik fork should make the best available version.

The features already available in ik_llama.cpp fork like MLA and custom tensor GPU offload make it difficult to go back to mainline llama.cpp for these big MoE's like R1 and V3-0324 until those experimental branches get merged in.

Logs

👇

👈 Testing Logs and Notes

Enable Explicit Hugepages

$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
3

# might take a few tries or require reboot or kernel boot param on your rig
$ echo 400000 | sudo tee -a /proc/sys/vm/nr_hugepages
400000

$ grep Huge /proc/meminfo
AnonHugePages:     96256 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:   400000
HugePages_Free:    400000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        819200000 kB

# this is about 390 GiB per NUMA Node which is enough for around Q4 or smaller quant
# it crashed on exit, and i had to manually clean up explicit huge pages something like this:
# $ echo 0 | sudo tee -a /proc/sys/vm/nr_hugepages
# $ echo 0 | sudo tee -a /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
# $ echo 0 | sudo tee -a /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
# $ echo 0 | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# $ mount | grep huge
# hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
# $ ls /dev/hugepages/
# $ sudo rm -rf /dev/hugepages/*
# $ echo 0 | sudo tee -a /proc/sys/vm/nr_hugepages

Compile

$ cmake -B ./build -DGGML_NATIVE=ON -DGGML_CUDA=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_INT8=OFF -DGGML_AMX_BF16=OFF -DGGML_NUMA_MIRROR=ON

...
/home/j/projects/llama.cpp/ggml/src/ggml-cpu/amx/mmq.cpp:2025:24: error: ‘struct ggml_tensor’ has no member named ‘data’
 2025 |         (float *) dst->data + 0 * N + nb_start, ldc)
...

## so try to comment everything out
$ grep -ri --include=CMakeLists.txt amx

[ 26%] Built target build_info
/usr/bin/ld: ../../bin/libggml-cpu.so: undefined reference to `ggml_backend_amx_buffer_type()'
collect2: error: ld returned 1 exit status
/usr/bin/ld: ../../bin/libggml-cpu.so: undefined reference to `ggml_backend_amx_buffer_type()'
gmake[2]: *** [examples/simple/CMakeFiles/llama-simple.dir/build.make:102: bin/llama-simple] Error 1
collect2: error: ld returned 1 exit status

## so remote all references from headers
$ grep -ri --include="*.h" amx

$ git diff ggml/include/ggml-cpu.h
diff --git a/ggml/include/ggml-cpu.h b/ggml/include/ggml-cpu.h
index f5e11f1e..368e294b 100644
--- a/ggml/include/ggml-cpu.h
+++ b/ggml/include/ggml-cpu.h
@@ -87,7 +87,7 @@ extern "C" {
     GGML_BACKEND_API int ggml_cpu_has_avx512_vbmi(void);
     GGML_BACKEND_API int ggml_cpu_has_avx512_vnni(void);
     GGML_BACKEND_API int ggml_cpu_has_avx512_bf16(void);
-    GGML_BACKEND_API int ggml_cpu_has_amx_int8   (void);
+    //GGML_BACKEND_API int ggml_cpu_has_amx_int8   (void);
     // ARM
     GGML_BACKEND_API int ggml_cpu_has_neon       (void);
     GGML_BACKEND_API int ggml_cpu_has_arm_fma    (void);

## now remove from source
$ grep -ri --include="*.cpp" ggml_cpu_has_amx

$ git diff ggml/src/ggml-cpu/ggml-cpu.cpp
diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cpp
index 09f8382b..d8e586dc 100644
--- a/ggml/src/ggml-cpu/ggml-cpu.cpp
+++ b/ggml/src/ggml-cpu/ggml-cpu.cpp
@@ -526,9 +526,9 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
         if (ggml_cpu_has_avx512_bf16()) {
             features.push_back({ "AVX512_BF16", "1" });
         }
-        if (ggml_cpu_has_amx_int8()) {
-            features.push_back({ "AMX_INT8", "1" });
-        }
+        //if (ggml_cpu_has_amx_int8()) {
+        //    features.push_back({ "AMX_INT8", "1" });
+        //}
         if (ggml_cpu_has_neon()) {
             features.push_back({ "NEON", "1" });
         }

# now remove more code
$ grep -ri --include="*.cpp" ggml_backend_amx | grep -v 'amx.cpp' | grep -v 'mmq.cpp'
ggml/src/ggml-cpu/ggml-cpu.cpp:        if (ggml_backend_amx_buffer_type()) {
ggml/src/ggml-cpu/ggml-cpu.cpp:            bufts.push_back(ggml_backend_amx_buffer_type());

$ git diff ggml/src/ggml-cpu/ggml-cpu.cpp
@@ -526,9 +526,9 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
         if (ggml_cpu_has_avx512_bf16()) {
             features.push_back({ "AVX512_BF16", "1" });
         }
-        if (ggml_cpu_has_amx_int8()) {
-            features.push_back({ "AMX_INT8", "1" });
-        }
+        //if (ggml_cpu_has_amx_int8()) {
+        //    features.push_back({ "AMX_INT8", "1" });
+        //}
         if (ggml_cpu_has_neon()) {
             features.push_back({ "NEON", "1" });
         }

# Okay now it compiles clean

Try it out

# Try a small enough model to fit into explicit huge pages
$ ./build/bin/llama-server \
    --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \
    --alias unsloth/DeepSeek-R1-UD_Q2_K_XL \
    --ctx-size 8192 \
    --parallel 1 \
    --threads 64 \
    --host 127.0.0.1 \
    --port 8080

load_tensors: loading model tensors, this can take a while... (mmap = true)
numa_set_preferred(0)
failed to open hugepage fd /dev/hugepages/llama-node0-0: 13 Permission denied
llama_model_load: error loading model: failed to open hugepage fd: Permission denied
llama_model_load_from_file_impl: failed to load model

## Now try again as root
$ sudo ./build/bin/llama-server \
    --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \
    --alias unsloth/DeepSeek-R1-UD_Q2_K_XL \
    --ctx-size 8192 \
    --parallel 1 \
    --threads 64 \
    --host 127.0.0.1 \
    --port 8080

load_tensors: loading model tensors, this can take a while... (mmap = true)
numa_set_preferred(0)
mmap(/dev/hugepages/llama-node0-0) desire=0x200000000000 size=1073741824 result=0x200000000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-1) desire=0x200040000000 size=1073741824 result=0x200040000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-2) desire=0x200080000000 size=1073741824 result=0x200080000000 is_new_mem[0]=yes
...
mmap(/dev/hugepages/llama-node0-44) desire=0x200b00000000 size=1073741824 result=0x200b00000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-45) desire=0x200b40000000 size=1073741824 result=0x200b40000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-46) desire=0x200b80000000 size=1073741824 result=0x200b80000000 is_new_mem[0]=yes
numa_set_preferred(1)
mmap(/dev/hugepages/llama-node1-0) desire=0x400000000000 size=1073741824 result=0x400000000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-1) desire=0x400040000000 size=1073741824 result=0x400040000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-2) desire=0x400080000000 size=1073741824 result=0x400080000000 is_new_mem[1]=yes
...
mmap(/dev/hugepages/llama-node1-44) desire=0x400b00000000 size=1073741824 result=0x400b00000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-45) desire=0x400b40000000 size=1073741824 result=0x400b40000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-46) desire=0x400b80000000 size=1073741824 result=0x400b80000000 is_new_mem[1]=yes
begin to copy from disk to mem ...
# i see about ~1.2GB/s read from NVMe drive during this time...
begin to copy from numa0 to numa1 ...
numa_set_preferred(0)
mmap(/dev/hugepages/llama-node0-47) desire=0x200bc0000000 size=1073741824 result=0x200bc0000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-48) desire=0x200c00000000 size=1073741824 result=0x200c00000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-49) desire=0x200c40000000 size=1073741824 result=0x200c40000000 is_new_mem[0]=yes
...
mmap(/dev/hugepages/llama-node0-91) desire=0x2016c0000000 size=1073741824 result=0x2016c0000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-92) desire=0x201700000000 size=1073741824 result=0x201700000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-93) desire=0x201740000000 size=1073741824 result=0x201740000000 is_new_mem[0]=yes
numa_set_preferred(1)
mmap(/dev/hugepages/llama-node1-47) desire=0x400bc0000000 size=1073741824 result=0x400bc0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-48) desire=0x400c00000000 size=1073741824 result=0x400c00000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-49) desire=0x400c40000000 size=1073741824 result=0x400c40000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-91) desire=0x4016c0000000 size=1073741824 result=0x4016c0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-92) desire=0x401700000000 size=1073741824 result=0x401700000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-93) desire=0x401740000000 size=1073741824 result=0x401740000000 is_new_mem[1]=yes
begin to copy from disk to mem ...
# again i see about ~1.3GB/s read from NVMe drive during this time...
begin to copy from numa0 to numa1 ...
numa_set_preferred(0)
mmap(/dev/hugepages/llama-node0-94) desire=0x201780000000 size=1073741824 result=0x201780000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-95) desire=0x2017c0000000 size=1073741824 result=0x2017c0000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-96) desire=0x201800000000 size=1073741824 result=0x201800000000 is_new_mem[0]=yes
...
mmap(/dev/hugepages/llama-node0-138) desire=0x202280000000 size=1073741824 result=0x202280000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-139) desire=0x2022c0000000 size=1073741824 result=0x2022c0000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-140) desire=0x202300000000 size=1073741824 result=0x202300000000 is_new_mem[0]=yes
numa_set_preferred(1)
mmap(/dev/hugepages/llama-node1-94) desire=0x401780000000 size=1073741824 result=0x401780000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-95) desire=0x4017c0000000 size=1073741824 result=0x4017c0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-96) desire=0x401800000000 size=1073741824 result=0x401800000000 is_new_mem[1]=yes
...
mmap(/dev/hugepages/llama-node1-138) desire=0x402280000000 size=1073741824 result=0x402280000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-139) desire=0x4022c0000000 size=1073741824 result=0x4022c0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-140) desire=0x402300000000 size=1073741824 result=0x402300000000 is_new_mem[1]=yes
begin to copy from disk to mem ...
...

# blah blah it keeps going slowly counting up

mmap(/dev/hugepages/llama-node1-211) desire=0x4034c0000000 size=1073741824 result=0x4034c0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-212) desire=0x403500000000 size=1073741824 result=0x403500000000 is_new_mem[1]=yes
begin to copy from disk to mem ...
begin to copy from numa0 to numa1 ...
load_tensors:   CPU_Mapped model buffer size = 47485.39 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 25569.12 MiB
...................................................................................................warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_per_seq = 8192
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 0.025
llama_context: n_ctx_per_seq (8192) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.49 MiB
init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
init:        CPU KV buffer size = 39040.00 MiB
llama_context: KV self size  = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB
llama_context:        CPU compute buffer size =  2218.01 MiB
llama_context: graph nodes  = 5025
llama_context: graph splits = 1
common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
thread_id = 00, node = 0, cpuid = 00
thread_id = 43, node = 0, cpuid = 43
thread_id = 48, node = 0, cpuid = 48
thread_id = 42, node = 0, cpuid = 42
thread_id = 30, node = 0, cpuid = 30
thread_id = 19, node = 0, cpuid = 19
thread_id = 37, node = 0, cpuid = 37
thread_id = 33, node = 0, cpuid = 33
thread_id = 32, node = 0, cpuid = 32
thread_id = 26, node = 0, cpuid = 26
thread_id = 40, node = 0, cpuid = 40
thread_id = 38, node = 0, cpuid = 38
thread_id = 36, node = 0, cpuid = 36
thread_id = 39, node = 0, cpuid = 39
thread_id = 25, node = 0, cpuid = 25
thread_id = 41, node = 0, cpuid = 41
thread_id = 31, node = 0, cpuid = 31
thread_id = 13, node = 0, cpuid = 13
thread_id = 05, node = 0, cpuid = 05
thread_id = 14, node = 0, cpuid = 14

thread_id = 28, node = 0, cpuid = 28
thread_id = 07, node = 0, cpuid = 07
thread_id = 29, node = 0, cpuid = 29
thread_id = 18, node = 0, cpuid = 18
thread_id = 08, node = 0, cpuid = 08
thread_id = 22, node = 0, cpuid = 22
thread_id = 60, node = 0, cpuid = 60
thread_id = 06, node = 0, cpuid = 06
thread_id = 17, node = 0, cpuid = 17
thread_id = 12, node = 0, cpuid = 12
thread_id = 51, node = 0, cpuid = 51
thread_id = 54, node = 0, cpuid = 54
thread_id = 63, node = 0, cpuid = 63
thread_id = 61, node = 0, cpuid = 61
thread_id = 56, node = 0, cpuid = 56
thread_id = 62, node = 0, cpuid = 62
thread_id = 55, node = 0, cpuid = 55
thread_id = 57, node = 0, cpuid = 57
thread_id = 59, node = 0, cpuid = 59
thread_id = 27, node = 0, cpuid = 27
thread_id = 52, node = 0, cpuid = 52
thread_id = 21, node = 0, cpuid = 21
thread_id = 10, node = 0, cpuid = 10
thread_id = 09, node = 0, cpuid = 09
thread_id = 16, node = 0, cpuid = 16
thread_id = 53, node = 0, cpuid = 53
thread_id = 01, node = 0, cpuid = 01
thread_id = 02, node = 0, cpuid = 02
thread_id = 11, node = 0, cpuid = 11
thread_id = 15, node = 0, cpuid = 15
thread_id = 49, node = 0, cpuid = 49
thread_id = 03, node = 0, cpuid = 03
thread_id = 23, node = 0, cpuid = 23
thread_id = 45, node = 0, cpuid = 45
thread_id = 34, node = 0, cpuid = 34
thread_id = 46, node = 0, cpuid = 46
thread_id = 50, node = 0, cpuid = 50
thread_id = 24, node = 0, cpuid = 24
thread_id = 47, node = 0, cpuid = 47
thread_id = 58, node = 0, cpuid = 58
thread_id = 44, node = 0, cpuid = 44
thread_id = 20, node = 0, cpuid = 20
thread_id = 04, node = 0, cpuid = 04
thread_id = 35, node = 0, cpuid = 35
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 8192
main: model loaded
main: chat template, chat_template:
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv  update_slots: all slots are idle

srv  log_server_r: request: GET /v1/models 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 13
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13, n_tokens = 13, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 13, n_tokens = 13
slot      release: id  0 | task 0 | stop processing: n_past = 676, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =     819.57 ms /    13 tokens (   63.04 ms per token,    15.86 tokens per second)
       eval time =   98072.72 ms /   664 tokens (  147.70 ms per token,     6.77 tokens per second)
      total time =   98892.29 ms /   677 tokens
srv  update_slots: all slots are idle

# ask it "Count from 1 to 10 in French" via my custom dchat.py client app

srv  log_server_r: request: GET /v1/models 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 13
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13, n_tokens = 13, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 13, n_tokens = 13
slot      release: id  0 | task 0 | stop processing: n_past = 676, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =     819.57 ms /    13 tokens (   63.04 ms per token,    15.86 tokens per second)
       eval time =   98072.72 ms /   664 tokens (  147.70 ms per token,     6.77 tokens per second)
      total time =   98892.29 ms /   677 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

$ grep Huge /proc/meminfo
AnonHugePages:  40169472 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:   400000
HugePages_Free:    181888
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        819200000 kB

$ sudo numastat -m -p $(pidof llama-server)

Per-node process memory usage (in MBs) for PID 2785429 (llama-server)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                    218112.00       218112.00       436224.00
Heap                        40.89            0.00           40.89
Stack                        0.05            0.00            0.05
Private                  39107.00            7.65        39114.65
----------------  --------------- --------------- ---------------
Total                   257259.94       218119.65       475379.59

Per-node system memory usage (in MBs):
Token Unaccepted not in hash table.
Token Unaccepted not in hash table.
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               771710.76       773987.20      1545697.96
MemFree                325179.17       151853.02       477032.19
MemUsed                446531.59       622134.18      1068665.77
SwapCached                  0.40            1.37            1.77
Active                  39294.56          210.40        39504.96
Inactive                  977.02       216733.16       217710.18
Active(anon)            39219.49           76.32        39295.80
Inactive(anon)              1.08           15.60           16.68
Active(file)               75.07          134.09          209.16
Inactive(file)            975.95       216717.56       217693.50
Unevictable                29.80            5.69           35.49
Mlocked                    21.01            5.69           26.70
Dirty                       0.02            0.00            0.02
Writeback                   0.00            0.00            0.00
FilePages                1063.30       216866.52       217929.82
Mapped                     53.62           60.35          113.98
AnonPages               39235.67           82.77        39318.44
Shmem                       8.97            7.82           16.79
KernelStack                46.81           37.95           84.77
PageTables                 83.63            3.31           86.95
SecPageTables               0.00            0.00            0.00
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                     2690.84         1905.19         4596.03
SReclaimable              665.92          184.16          850.08
SUnreclaim               2024.92         1721.03         3745.95
AnonHugePages           39188.00           40.00        39228.00
ShmemHugePages              0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00
FileHugePages               0.00            0.00            0.00
FilePmdMapped               0.00            0.00            0.00
HugePages_Total        400000.00       400000.00       800000.00
HugePages_Free         181888.00       181888.00       363776.00
HugePages_Surp              0.00            0.00            0.00
KReclaimable              665.92          184.16          850.08

Observations

It seems odd that the 64 threads it chose are all in a single NUMA node instead of distributed equally across both?

This system is set SNC=Disable which is roughly equivilent of AMD Epyc dual socket configured to NPS1.

$ lscpu | grep NUMA
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-127,256-383
NUMA node1 CPU(s):                    128-255,384-511

☝️

[ikawrakow]

@usrlocalben

are they documented or can you point me towards a description?

Sorry, I'm not the documentation type of guy (and I don't have 1000+ contributors as this project does to help with that😄), so information is spread around the various PRs, and there isn't a single place that explains all the features. I'm very grateful to @ubergarm for putting together more organized and easily accessible information.

Concerning the -mla 3 -fa -fmoe options, in a nutshell:

• -fmoe: this fuses experts ffn_up and ffn_gate matrix multiplications, along with the subsequent unary op (SILU for the DeepSeek MoE models) into a single op. On CUDA, it also fuses the ffn_down matrix multiplication with the result of unary((ffn_up*X)*(ffn_gate*X). On CUDA this leads to nearly a 2X performance improvement for PP, and 20-30% for TG. On the CPU the effect is more modest, but not negligible.
• Vanilla MLA (as implemented by @fairydreaming here in PR Optimized DeepSeek V2/V3 implementation (MLA) #11446) leads to better TG performance at the expense of a massive loss in PP performance. The -mla 3 option combined with Flash Attention (-fa) further improves TG performance while recovering PP performance.

what are the context-size constraints in the mixed gpu/cpu config?

I think @ubergarm answered that above. For more background, mla = 3 only requires compressed K-cache, using 576*61 (MLA K-head size times number of layers) entries per token in the cache. Hence, if you quantize the cache with Q8_0, you only need ~5.7 GiB for a context of 163k tokens. The required compute buffer is larger, but its size can be controlled via another command line option (-amb). Depending on the bpw used to quantize attention and shared experts tensors, you may be able to arrive at a context of 65k on a single 24 GB GPU with a hybrid GPU/CPU setup.

It has taken quite a bit of experimentation and gradual improvements to arrive at the current state. Relevent PRs are:
ikawrakow/ik_llama.cpp#283, ikawrakow/ik_llama.cpp#277, ikawrakow/ik_llama.cpp#273 ikawrakow/ik_llama.cpp#260 ikawrakow/ik_llama.cpp#253 ikawrakow/ik_llama.cpp#248 ikawrakow/ik_llama.cpp#247 ikawrakow/ik_llama.cpp#243 ikawrakow/ik_llama.cpp#241 ikawrakow/ik_llama.cpp#240 ikawrakow/ik_llama.cpp#237 ikawrakow/ik_llama.cpp#235 ikawrakow/ik_llama.cpp#234 ikawrakow/ik_llama.cpp#229 ikawrakow/ik_llama.cpp#205 ikawrakow/ik_llama.cpp#200 ikawrakow/ik_llama.cpp#195 ikawrakow/ik_llama.cpp#188

[ubergarm]

@wkgcass

Hi, could you share the output of your lscpu? On my platform it shows:

Yeah this dual socket intel xeon 6980P is configured in BIOS to SNC=Disable to present two numa nodes (one per socket) instead of the default six total.

$ lscpu | grep NUMA
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-127,256-383
NUMA node1 CPU(s):                    128-255,384-511

taskset -c 0-7,24-31

Ahh yes, I understand, the design assumes the user will manually assign appropriate cpu cores via taskset to split between the NUMA nodes. I'll give it another try and see how it goes.

Thanks!

[ubergarm]

@wkgcass

Okay, I tried again usting taskset to pin CPU threads distributed across both NUMA nodes.

Detailed Logs

Given the Intel Xeon 6980P is physically broken into 3x compute dies with physical cores 43+43+42 physical cores I decided to place 24 threads onto each compute die across both CPU sockets hoping to improve cache chance maybe... 144 total threads is nice because it is divisible by 3 and also by 8. Though in actual usage I would probably want to use --threads and --threads-batch to use less threads for token generation and more threads for prompt processing. But trying to keep it a little more simple for initial testing.

sudo taskset -c 0-23,43-66,86-109,128-151,171-194,214-237 \
./build/bin/llama-server \
    --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \
    --alias unsloth/DeepSeek-R1-UD_Q2_K_XL \
    --ctx-size 8192 \
    --parallel 1 \
    --threads 144 \
    --host 127.0.0.1 \
    --port 8080

Current power profile is: performance
Current THP enabled and defrag configs are:
[always] madvise never
[always] defer defer+madvise madvise never
Set numa balancing to be:
0
build: 4888 (f86568f8) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
system info: n_threads = 144, n_threads_batch = 144, total_threads = 512

system_info: n_threads = 144 (n_threads_batch = 144) / 512 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI
2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 511
main: loading model
.
.
.
llama_model_loader: - type  f32:  361 tensors
llama_model_loader: - type q2_K:  171 tensors
llama_model_loader: - type q3_K:    3 tensors
llama_model_loader: - type q4_K:  306 tensors
llama_model_loader: - type q6_K:  184 tensors
.
.
.
load_tensors: loading model tensors, this can take a while... (mmap = true)
numa_set_preferred(0)
mmap(/dev/hugepages/llama-node0-0) desire=0x200000000000 size=1073741824 result=0x200000000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-1) desire=0x200040000000 size=1073741824 result=0x200040000000 is_new_mem[0]=yes
mmap(/dev/hugepages/llama-node0-2) desire=0x200080000000 size=1073741824 result=0x200080000000 is_new_mem[0]=yes
.
.
.
mmap(/dev/hugepages/llama-node1-210) desire=0x403480000000 size=1073741824 result=0x403480000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-211) desire=0x4034c0000000 size=1073741824 result=0x4034c0000000 is_new_mem[1]=yes
mmap(/dev/hugepages/llama-node1-212) desire=0x403500000000 size=1073741824 result=0x403500000000 is_new_mem[1]=yes
begin to copy from disk to mem ...
begin to copy from numa0 to numa1 ...
load_tensors:   CPU_Mapped model buffer size = 47485.39 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 25569.12 MiB
...................................................................................................warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
warning: munmap failed: Invalid argument
.
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_per_seq = 8192
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 0.025
llama_context: n_ctx_per_seq (8192) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.49 MiB
init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
init:        CPU KV buffer size = 39040.00 MiB
llama_context: KV self size  = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB
llama_context:        CPU compute buffer size =  2218.01 MiB
llama_context: graph nodes  = 5025
llama_context: graph splits = 1
.
.
.
thread_id = 00, node = 0, cpuid = 00
thread_id = 96, node = 1, cpuid = 171
thread_id = 127, node = 1, cpuid = 221
thread_id = 122, node = 1, cpuid = 216
thread_id = 05, node = 0, cpuid = 05
thread_id = 38, node = 0, cpuid = 57
thread_id = 39, node = 0, cpuid = 58
thread_id = 111, node = 1, cpuid = 186
thread_id = 35, node = 0, cpuid = 54
thread_id = 98, node = 1, cpuid = 173
thread_id = 31, node = 0, cpuid = 50
thread_id = 104, node = 1, cpuid = 179
thread_id = 72, node = 1, cpuid = 128
thread_id = 36, node = 0, cpuid = 55
thread_id = 43, node = 0, cpuid = 62
thread_id = 33, node = 0, cpuid = 52
thread_id = 09, node = 0, cpuid = 09
thread_id = 85, node = 1, cpuid = 141
thread_id = 25, node = 0, cpuid = 44
thread_id = 60, node = 0, cpuid = 98
thread_id = 50, node = 0, cpuid = 88
thread_id = 70, node = 0, cpuid = 108
thread_id = 52, node = 0, cpuid = 90
thread_id = 91, node = 1, cpuid = 147
thread_id = 78, node = 1, cpuid = 134
thread_id = 55, node = 0, cpuid = 93
thread_id = 82, node = 1, cpuid = 138
thread_id = 53, node = 0, cpuid = 91
thread_id = 65, node = 0, cpuid = 103
thread_id = 138, node = 1, cpuid = 232
thread_id = 101, node = 1, cpuid = 176
thread_id = 118, node = 1, cpuid = 193
thread_id = 119, node = 1, cpuid = 194
thread_id = 113, node = 1, cpuid = 188
thread_id = 97, node = 1, cpuid = 172
thread_id = 121, node = 1, cpuid = 215
thread_id = 114, node = 1, cpuid = 189
thread_id = 140, node = 1, cpuid = 234
thread_id = 128, node = 1, cpuid = 222
thread_id = 124, node = 1, cpuid = 218
thread_id = 79, node = 1, cpuid = 135
thread_id = 108, node = 1, cpuid = 183
thread_id = 116, node = 1, cpuid = 191
thread_id = 115, node = 1, cpuid = 190
thread_id = 120, node = 1, cpuid = 214
thread_id = 92, node = 1, cpuid = 148
thread_id = 73, node = 1, cpuid = 129
thread_id = 18, node = 0, cpuid = 18
thread_id = 19, node = 0, cpuid = 19
thread_id = 87, node = 1, cpuid = 143
thread_id = 22, node = 0, cpuid = 22
thread_id = 90, node = 1, cpuid = 146
thread_id = 88, node = 1, cpuid = 144
thread_id = 13, node = 0, cpuid = 13
thread_id = 20, node = 0, cpuid = 20
thread_id = 131, node = 1, cpuid = 225
thread_id = 16, node = 0, cpuid = 16
thread_id = 49, node = 0, cpuid = 87
thread_id = 83, node = 1, cpuid = 139
thread_id = 77, node = 1, cpuid = 133
thread_id = 12, node = 0, cpuid = 12
thread_id = 01, node = 0, cpuid = 01
thread_id = 117, node = 1, cpuid = 192
thread_id = 62, node = 0, cpuid = 100
thread_id = 26, node = 0, cpuid = 45
thread_id = 04, node = 0, cpuid = 04
thread_id = 23, node = 0, cpuid = 23
thread_id = 15, node = 0, cpuid = 15
thread_id = 03, node = 0, cpuid = 03
thread_id = 45, node = 0, cpuid = 64
thread_id = 40, node = 0, cpuid = 59
thread_id = 71, node = 0, cpuid = 109
thread_id = 28, node = 0, cpuid = 47
thread_id = 24, node = 0, cpuid = 43
thread_id = 141, node = 1, cpuid = 235
thread_id = 143, node = 1, cpuid = 237
thread_id = 133, node = 1, cpuid = 227
thread_id = 132, node = 1, cpuid = 226
thread_id = 137, node = 1, cpuid = 231
thread_id = 93, node = 1, cpuid = 149
thread_id = 109, node = 1, cpuid = 184
thread_id = 107, node = 1, cpuid = 182
thread_id = 105, node = 1, cpuid = 180
thread_id = 08, node = 0, cpuid = 08
thread_id = 68, node = 0, cpuid = 106
thread_id = 57, node = 0, cpuid = 95
thread_id = 48, node = 0, cpuid = 86
thread_id = 42, node = 0, cpuid = 61
thread_id = 86, node = 1, cpuid = 142
thread_id = 134, node = 1, cpuid = 228
thread_id = 81, node = 1, cpuid = 137
thread_id = 125, node = 1, cpuid = 219
thread_id = 123, node = 1, cpuid = 217
thread_id = 02, node = 0, cpuid = 02
thread_id = 89, node = 1, cpuid = 145
thread_id = 135, node = 1, cpuid = 229
thread_id = 75, node = 1, cpuid = 131
thread_id = 136, node = 1, cpuid = 230
thread_id = 76, node = 1, cpuid = 132
thread_id = 95, node = 1, cpuid = 151
thread_id = 74, node = 1, cpuid = 130
thread_id = 94, node = 1, cpuid = 150
thread_id = 110, node = 1, cpuid = 185
thread_id = 112, node = 1, cpuid = 187
thread_id = 100, node = 1, cpuid = 175
thread_id = 46, node = 0, cpuid = 65
thread_id = 106, node = 1, cpuid = 181
thread_id = 103, node = 1, cpuid = 178
thread_id = 139, node = 1, cpuid = 233
thread_id = 54, node = 0, cpuid = 92
thread_id = 10, node = 0, cpuid = 10
thread_id = 84, node = 1, cpuid = 140
thread_id = 44, node = 0, cpuid = 63
thread_id = 142, node = 1, cpuid = 236
thread_id = 59, node = 0, cpuid = 97
thread_id = 30, node = 0, cpuid = 49
thread_id = 56, node = 0, cpuid = 94
thread_id = 47, node = 0, cpuid = 66
thread_id = 61, node = 0, cpuid = 99
thread_id = 06, node = 0, cpuid = 06
thread_id = 34, node = 0, cpuid = 53
thread_id = 67, node = 0, cpuid = 105
thread_id = 11, node = 0, cpuid = 11
thread_id = 27, node = 0, cpuid = 46
thread_id = 130, node = 1, cpuid = 224
thread_id = 41, node = 0, cpuid = 60
thread_id = 37, node = 0, cpuid = 56
thread_id = 58, node = 0, cpuid = 96
thread_id = 126, node = 1, cpuid = 220
thread_id = 32, node = 0, cpuid = 51
thread_id = 129, node = 1, cpuid = 223
thread_id = 29, node = 0, cpuid = 48
thread_id = 64, node = 0, cpuid = 102
thread_id = 66, node = 0, cpuid = 104
thread_id = 69, node = 0, cpuid = 107
thread_id = 80, node = 1, cpuid = 136
thread_id = 63, node = 0, cpuid = 101
thread_id = 07, node = 0, cpuid = 07
thread_id = 102, node = 1, cpuid = 177
thread_id = 17, node = 0, cpuid = 17
thread_id = 51, node = 0, cpuid = 89
thread_id = 14, node = 0, cpuid = 14
thread_id = 21, node = 0, cpuid = 21
thread_id = 99, node = 1, cpuid = 174
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 8192
main: model loaded

> Count from 1 to 10 in French.

prompt eval time =    2099.20 ms /    20 tokens (  104.96 ms per token,     9.53 tokens per second)
       eval time =   98422.62 ms /   454 tokens (  216.79 ms per token,     4.61 tokens per second)
      total time =  100521.83 ms /   474 tokens

> Now give pronunciation tips.

prompt eval time =    4003.26 ms /   105 tokens (   38.13 ms per token,    26.23 tokens per second)
       eval time =  254223.50 ms /  1124 tokens (  226.18 ms per token,     4.42 tokens per second)
      total time =  258226.76 ms /  1229 tokens

---

$ sudo numastat -m -p $(pidof llama-server)

Per-node process memory usage (in MBs) for PID 2203149 (llama-server)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                    218112.00       218112.00       436224.00
Heap                        34.71            1.92           36.63
Stack                        0.04            0.01            0.05
Private                     49.59        39059.21        39108.80
----------------  --------------- --------------- ---------------
Total                   218196.34       257173.14       475369.48

Per-node system memory usage (in MBs):
Token Unaccepted not in hash table.
Token Unaccepted not in hash table.
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               771710.76       773987.20      1545697.96
MemFree                366143.98       113368.32       479512.31
MemUsed                405566.78       660618.88      1066185.66
SwapCached                  1.05            1.79            2.83
Active                    173.04        39195.57        39368.61
Inactive                   58.15       216148.67       216206.82
Active(anon)              133.92        39124.73        39258.65
Inactive(anon)             13.63           12.07           25.70
Active(file)               39.12           70.84          109.96
Inactive(file)             44.52       216136.60       216181.12
Unevictable                29.86            5.50           35.36
Mlocked                    21.07            5.50           26.57
Dirty                       0.55            0.96            1.52
Writeback                   0.00            0.00            0.00
FilePages                 102.65       216214.85       216317.50
Mapped                     44.52           93.51          138.03
AnonPages                 158.73        39135.53        39294.25
Shmem                      14.86            0.12           14.98
KernelStack                46.34           39.69           86.03
PageTables                  7.54           80.62           88.17
SecPageTables               0.00            0.00            0.00
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                     2308.57         2309.95         4618.52
SReclaimable              198.09          602.74          800.84
SUnreclaim               2110.47         1707.21         3817.68
AnonHugePages             102.00        39100.00        39202.00
ShmemHugePages              0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00
FileHugePages               0.00            0.00            0.00
FilePmdMapped               0.00            0.00            0.00
HugePages_Total        400000.00       400000.00       800000.00
HugePages_Free         181888.00       181888.00       363776.00
HugePages_Surp              0.00            0.00            0.00
KReclaimable              198.09          602.74          800.84

vproxy-tools btop during llama-bench pp

So it does seem to be working copying entire model weights into Explicit Huge Pages twice (once per NUMA node) and mapping thread id's correctly for NUMA memory node <-> NUMA cpu node.

I'll try to run some llama-sweep-bench llama-bench to compare against mainline soon!

[ubergarm]

@wkgcass

Okay, ran some llama-bench to compare your PR running on two NUMA nodes vs mainline running on two NUMA nodes (via numactl interleave=all llama-bench --numa distribute which generally gives similar or slightly higher pp and lower tg compared to numactl -N 0 -m 0 llama-bench --numa numactl single NUMA node). I believe it is working, when I checked with --verbose it prints out all the Explicit Huge Pages as already existing and maps the thread id's.

:oof: no llama-sweep-bench and even llama-bench support for -gp has not been merged in yet ... so results are not as interesting to compare for tg at given prompt lengths... might be able to cherry-pick those into this numa node test and rebuild, yes:

Add better llama-bench support

Add fairydreaming's PR11126

$ git remote add [email protected]:fairydreaming/llama.cpp.git
$ git fetch fairydreaming
$ git merge fairydreaming/llama-bench-gp
$ git log --pretty=format:"%h%x09%an%x09%ad%x09%s" | head -n7

6d0034ae        John W. Leimgruber III  Fri Apr 11 17:56:53 2025 +0000  Merge remote-tracking branch 'fairydreaming/llama-bench-gp' into HEAD
f86568f8        wkgcass Fri Mar 14 14:16:21 2025 +0800  auto numa selection and mem binding
af599819        wkgcass Mon Mar 10 01:10:33 2025 +0800  compile with src symbols
93142867        wkgcass Mon Mar 10 01:08:28 2025 +0800  a rough implementation of GGML_NUMA_MIRROR
2f5f09b8        wkgcass Mon Mar 10 00:52:23 2025 +0800  add GGML_NUMA_MIRROR macro and modify tensor def
6773189c        wkgcass Sun Mar 9 21:52:34 2025 +0800   use tensor_data(t) and tensor_set_data(t,data)
84d54755        Georgi Gerganov Thu Mar 13 19:08:07 2025 +0200  llama : fix Gemma3 SWA KV cache shift (#12373)

# be sure to re-build, do this for both the test PR and mainline

llama-bench

Okay, now we can do some more meaningful benchmarks, though not as thorough as llama-sweep-bench.

Detailed Logs

vproxy-tools/will-force-push@f86568f8

./build/bin/llama-bench \
    --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \
    -p 512,2048,4096,8192 -n 0 \
    -gp 512,64 \
    -gp 2048,64 \
    -gp 4096,64 \
    -gp 8192,64 \
    -r 2 \
    --threads 144

...
mmap(/dev/hugepages/llama-node1-210) desire=0x403480000000 size=1073741824 result=0x403480000000 is_new_mem[1]=no
mmap(/dev/hugepages/llama-node1-211) desire=0x4034c0000000 size=1073741824 result=0x4034c0000000 is_new_mem[1]=no
mmap(/dev/hugepages/llama-node1-212) desire=0x403500000000 size=1073741824 result=0x403500000000 is_new_mem[1]=no
load_tensors:   CPU_Mapped model buffer size = 47485.39 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 47681.52 MiB
load_tensors:   CPU_Mapped model buffer size = 25569.12 MiB
...

$ sudo numastat -m -p $(pidof llama-bench)
Per-node process memory usage (in MBs) for PID 2210448 (llama-bench)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                    218112.00       218112.00       436224.00
Heap                        38.60            0.00           38.60
Stack                        0.03            0.00            0.03
Private                  41751.14          548.35        42299.49
----------------  --------------- --------------- ---------------
Total                   259901.77       218660.35       478562.12

Per-node system memory usage (in MBs):
Token Unaccepted not in hash table.
Token Unaccepted not in hash table.
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               771710.76       773987.20      1545697.96
MemFree                324396.86       152126.96       476523.82
MemUsed                447313.90       621860.24      1069174.14
SwapCached                  1.05            1.80            2.85
Active                  41889.79          692.11        42581.89
Inactive                   64.89       216149.27       216214.16
Active(anon)            41846.52          616.93        42463.45
Inactive(anon)             13.63           12.07           25.70
Active(file)               43.27           75.18          118.45
Inactive(file)             51.27       216137.20       216188.46
Unevictable                29.86            5.50           35.36
Mlocked                    21.07            5.50           26.57
Dirty                       0.32            0.25            0.57
Writeback                   0.00            0.00            0.00
FilePages                 113.58       216219.80       216333.38
Mapped                     47.22           92.06          139.29
AnonPages               41870.74          627.66        42498.40
Shmem                      14.86            0.12           14.98
KernelStack                40.61           37.44           78.05
PageTables                 86.40            3.88           90.28
SecPageTables               0.00            0.00            0.00
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                     2301.81         2310.16         4611.97
SReclaimable              198.35          602.73          801.08
SUnreclaim               2103.46         1707.43         3810.89
AnonHugePages           41806.00          594.00        42400.00
ShmemHugePages              0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00
FileHugePages               0.00            0.00            0.00
FilePmdMapped               0.00            0.00            0.00
HugePages_Total        400000.00       400000.00       800000.00
HugePages_Free         181888.00       181888.00       363776.00
HugePages_Surp              0.00            0.00            0.00
KReclaimable              198.35          602.73          801.08

RESULTS

model	size	params	backend	threads	test	t/s
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	pp512	39.77 ± 0.11
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	pp2048	34.65 ± 0.10
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	pp4096	32.46 ± 0.10
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	pp8192	26.24 ± 0.01
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	tg64@pp512	5.22 ± 0.00
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	tg64@pp2048	4.79 ± 0.00
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	tg64@pp4096	4.31 ± 0.01
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	tg64@pp8192	3.51 ± 0.00

build: 6d0034ae (4892) but really just commit: build: f86568f8 (4888) plus -gp stuff

---

To keep it as close to apples to apples as possible, I used 2x NUMA nodes with best best practice of numactl --interleave=all llama-bench -numa numactl, disable numa balancing, keep the exact same CPU cores (oops: maybe should have used numactl before taskset as it seems to not be pinned), and disable mmap after setting THP/defrag to always and confirm THPs in use.

llama.cpp/main@6a72b5c6

# first merge in fairydreaming's PR11126 to enable `llama-bench -gp` like shown above

sudo taskset -c 0-23,43-66,86-109,128-151,171-194,214-237 \
numactl --interleave=all \
./build/bin/llama-bench \
    --mmap 0 \
    --model /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf \
    -p 512,2048,4096,8192 -n 0 \
    -gp 512,64 \
    -gp 2048,64 \
    -gp 4096,64 \
    -gp 8192,64 \
    -r 2 \
    --threads 144 \
    --numa distribute

# ideally AnonHugePages will contain all of model weights
du -c /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-000*.gguf
48630184        /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00001-of-00005.gguf
48825900        /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00002-of-00005.gguf
48825900        /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00003-of-00005.gguf
48825900        /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00004-of-00005.gguf
26182788        /mnt/ai/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL/DeepSeek-R1-UD-Q2_K_XL-00005-of-00005.gguf
221290672       total

$ grep Huge /proc/meminfo

AnonHugePages:  243369984 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

$ sudo numastat -m -p $(pidof llama-bench)

Per-node process memory usage (in MBs) for PID 2258047 (llama-bench)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        51.59           29.60           81.19
Stack                        0.02            0.02            0.04
Private                 118188.26       118189.91       236378.16
----------------  --------------- --------------- ---------------
Total                   118239.87       118219.52       236459.39

Per-node system memory usage (in MBs):
Token Unaccepted not in hash table.
Token Unaccepted not in hash table.
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               771710.76       773987.20      1545697.96
MemFree                538940.60       542205.94      1081146.54
MemUsed                232770.16       231781.27       464551.43
SwapCached                  1.04            1.80            2.84
Active                 118342.58       118401.17       236743.75
Inactive               108166.62       108170.73       216337.35
Active(anon)           118286.95       118299.65       236586.60
Inactive(anon)             13.61           12.05           25.66
Active(file)               55.63          101.52          157.15
Inactive(file)         108153.02       108158.68       216311.70
Unevictable                29.86            5.50           35.36
Mlocked                    21.07            5.50           26.57
Dirty                       0.31            0.15            0.46
Writeback                   0.00            0.00            0.00
FilePages              108222.00       108273.25       216495.25
Mapped                     72.14           91.27          163.41
AnonPages              118316.88       118304.77       236621.66
Shmem                       9.23            5.75           14.98
KernelStack                39.45           38.52           77.97
PageTables                234.76          233.93          468.69
SecPageTables               0.00            0.00            0.00
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                     2730.74         1859.49         4590.23
SReclaimable              636.22          142.99          779.21
SUnreclaim               2094.52         1716.50         3811.02
AnonHugePages          118250.00       118262.00       236512.00
ShmemHugePages              0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00
FileHugePages               0.00            0.00            0.00
FilePmdMapped               0.00            0.00            0.00
HugePages_Total             0.00            0.00            0.00
HugePages_Free              0.00            0.00            0.00
HugePages_Surp              0.00            0.00            0.00
KReclaimable              636.22          142.99          779.21

RESULTS

model	size	params	backend	threads	mmap	test	t/s
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	pp512	39.77 ± 0.05
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	pp2048	34.38 ± 0.20
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	pp4096	31.53 ± 0.01
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	pp8192	25.66 ± 0.12
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	tg64@pp512	5.11 ± 0.00
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	tg64@pp2048	4.85 ± 0.03
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	tg64@pp4096	4.51 ± 0.00
deepseek2 671B Q2_K - Medium	211.03 GiB	671.03 B	CPU	144	0	tg64@pp8192	3.81 ± 0.01

build: 6a72b5c6 (5121) but really just commit: build: 578754b plus -gp stuff

mainline btop during llama-bench tg

*Note that I forgot llama-bench -numa distribute does not honor taskset... so threads are all over the place here...

Results

Conclusion

Well, surprisingly the vproxy-tools test case it doesn't seem much different from mainline. Perhaps because only using 144 out of 256 physical cores isn't saturating the ~512 GB/s RAM bandwidth per socket? Perhaps because my merging in the extra llama-bench -gp feature messed something up? Perhaps something because the -numa distribute case llama-bench does not honor taskset and letting the threads run un-pinned was better than how I pinned them?

Sorry, not the most exciting results lol...

Well, just for reference, while it is not apples to apples, you can see performance on ik_llama.cpp benchmark results are much faster using a single NUMA node with a bigger quant (V3-0324 vs R1 also).

Might be best re test this multi-numa branch after MLA support is merged in otherwise anything over 8k context is just too painfully slow to test.

[saood06]

:oof: no llama-sweep-bench and even llama-bench support for -gp has not been merged in yet

I always preferred llama-batched-bench at a batch size of 1 over -gp as it shows you both PP and TG (only downside is it does not support -r and is equivalent to running bench with -r 1)

[usrlocalben]

I was curious how this patch performs on recent builds. I rebased it to bring it up to date. It runs fine and gives valid output, but it runs at 1t/s. I observe the thread_id = {}, node = {}, cpuid = {} log message is emitted persistently during token-gen, so I assume something has changed wrt. threading/execution model.

[usrlocalben]

Upon reflection I thought there must be another reason for the throughput trouble. It turned out to be related to clang/openmp/cmake badness. Here's a few runs below. I suppose with GPU offloading and a few months of incremental changes the difference has narrowed.

Kimi K2 Q4_K_M
2S EPYC 9115, w/attention, shared tensors etc. on RTX 8000

standard    numa=mirror      mainline
prompt eval time =   55077.81 ms /  1958 tokens (   28.13 ms per token,    35.55 tokens per second)
       eval time =  180216.52 ms /  2115 tokens (   85.21 ms per token,    11.74 tokens per second)
      total time =  235294.33 ms /  4073 tokens

standard    numa=distribute  mainline
prompt eval time =   55145.00 ms /  1958 tokens (   28.16 ms per token,    35.51 tokens per second)
       eval time =  279231.03 ms /  3153 tokens (   88.56 ms per token,    11.29 tokens per second)
      total time =  334376.03 ms /  5111 tokens

standard    numa=off/NPS0    mainline
prompt eval time =   54869.57 ms /  1958 tokens (   28.02 ms per token,    35.68 tokens per second)
       eval time =  307056.96 ms /  3455 tokens (   88.87 ms per token,    11.25 tokens per second)
      total time =  361926.53 ms /  5413 tokens

standard    numa=off/NPS0    ik_llama
prompt eval time =   25175.35 ms /  1969 tokens (   12.79 ms per token,    78.21 tokens per second)
       eval time =  244944.73 ms /  2873 runs   (   85.26 ms per token,    11.73 tokens per second)
      total time =  270120.08 ms

bonus: speculative decoding + mirror (token output content is code)

prompt eval time =   55197.71 ms /  1958 tokens (   28.19 ms per token,    35.47 tokens per second)
       eval time =  135731.23 ms /  2017 tokens (   67.29 ms per token,    14.86 tokens per second)
      total time =  190928.95 ms /  3975 tokens

[sultanqasim]

Have you tried NUMA mirroring with smaller dense models and no GPU offload? The test you did is quite apples to oranges compared to what the original tests were.