ggml : implement REGLU/GEGLU/SWIGLU ops #14158

CISC · 2025-06-12T23:24:57Z

Implement REGLU/GEGLU/SWIGLU ops to avoid unnecessary tensor duplications and a little more efficient execution by combining ops in one.

Only CPU and CUDA right now, help needed to complete other backends!

ggerganov

I missed that these ops change the shape of the input tensor.

I think it would be better to introduce:

enum ggml_glu_op {
    GGML_GLU_OP_REGLU,
    GGML_GLU_OP_GEGLU,
    GGML_GLU_OP_SWIGLU,
};

// similar to ggml_unary()
GGML_API struct ggml_tensor * ggml_glu(
        struct ggml_context * ctx,
         struct ggml_tensor * a,
           enum ggml_glu_op   op);

// these simply call ggml_glu()
GGML_API struct ggml_tensor * ggml_reglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

GGML_API struct ggml_tensor * ggml_geglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

GGML_API struct ggml_tensor * ggml_swiglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

ggml/include/ggml.h

ggerganov

Hope we don't forget to implement these in the rest of the backends.

Adding @JohannesGaessler for review of the CUDA changes.

ggerganov · 2025-06-13T08:31:31Z

Only CPU and CUDA right now, help needed to complete other backends!

Yes, let's add the rest of the backends first before merging. At least Metal and Vulkan.

ggml/src/ggml-cuda/unary.cu

JohannesGaessler · 2025-06-13T09:14:48Z

More generally, I've been thinking that it would be useful to have something like a backend-specific graph optimization step in ggml. That way you could do things like fuse tensors only if the fused tensor is supported by the backend and only if using it makes sense given the tensor shapes.

CISC · 2025-06-13T10:29:17Z

Only CPU and CUDA right now, help needed to complete other backends!

Yes, let's add the rest of the backends first before merging. At least Metal and Vulkan.

Any suggestions on who could help with that?

ggml-ci

ngxson · 2025-06-13T13:14:56Z

ggml/include/ggml.h

+            struct ggml_context * ctx,
+            struct ggml_tensor  * a);
+
+    GGML_API struct ggml_tensor * ggml_swiglu(


just want to note that I have been observing one variants of swiglu. it's used by ultravox, which sigmoid the second half of the vector instead of the first half

Oh, interesting, worth adding a parameter for, or best just handling in conversion?
https://huggingface.co/fixie-ai/ultravox-v0_5-llama-3_3-70b/blob/main/ultravox_model.py#L701-L704

I think it would be nice to have a param since the GGUFs are already on the internet. Haven't thought about permuting the FFN up tensor before, nice suggestion

Added swapped variants.

@ggerganov I didn't dare update metal code, so needs to be implemented there too. :)

JohannesGaessler · 2025-06-13T13:29:00Z

@0cc4m @jeffbolznv are either of you interested in a Vulkan implementation?

0cc4m · 2025-06-13T13:42:21Z

I can look into it tomorrow.

ggml-ci

ggml/src/ggml-cuda/unary.cu

CISC · 2025-06-13T21:38:33Z

Hmmm, it just occurred to me that we should be able to (now that I pass along a pointer to the gate separately) perform these ops on models with separate ffn_up/gate tensors too by conditionally setting src[1].

ngxson · 2025-06-13T21:59:11Z

ggml/include/ggml.h

+            struct ggml_context * ctx,
+            struct ggml_tensor  * a);
+
+    GGML_API struct ggml_tensor * ggml_geglu(


Tbh I don't even know why geglu was added in the first place. It doesn't seem to be used by any models. And to make matter worse, the PR where it was added has no useful description: #14074

So I wonder if we actually need to implement it as a kernel. The current kernel use tanh approximation, but in practice, there can be many different approximations for gelu op.

Nvm, see: https://github.com/ggml-org/llama.cpp/pull/14014/files#r2146203459

I've seen several, and in fact we already support a few (Gemma, DeepSeekV1, Jina-Bert and T5), it's just that the gate is split (some at conversion because we didn't have the op).

So I wonder if we actually need to implement it as a kernel. The current kernel use tanh approximation, but in practice, there can be many different approximations for gelu op.

It's pretty easy adding different GLU ops (and in CUDA I even reuse the original op), adding GEGLU_ERF if necessary shouldn't be a problem.

0cc4m · 2025-06-14T10:14:36Z

I implemented Vulkan shaders for the new ops.

qnixsynapse · 2025-06-14T13:43:41Z

Interesting.. I tried implementing for SYCL, saw little improvement. When I saw the graph logs, it wasn't using the fused kernels for llama 3.2 3B.

[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_gate.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_gate.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_silu: dst='ffn_silu-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_silu done
[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_up.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_up.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_mul: dst='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='ffn_silu-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src1='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_mul done
[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='blk.19.ffn_down.weight':type=f16;ne=[8192, 3072, 1, 1];nb=[2, 16384, 50331648, 50331648]	src1='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='blk.19.ffn_down.weight':type=f16;ne=[8192, 3072, 1, 1];nb=[2, 16384, 50331648, 50331648]	src1='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_add: dst='l_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src1='ffn_inp-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_add done

I am using from this branch.

CISC · 2025-06-14T13:50:21Z

Interesting.. I tried implementing for SYCL, saw little improvement. When I saw the graph logs, it wasn't using the fused kernels for llama 3.2 3B.

That's normal, llama 3.2 doesn't have a single up+gate does it?

qnixsynapse · 2025-06-14T13:55:44Z

IIRC, it has SWIGLU. But I didn't check if it is using a single up+gate or not.

Edit: Nevermind, Need to check #14181 I guess

CISC · 2025-06-14T14:12:01Z

Edit: Nevermind, Need to check #14181 I guess

Yep. :)

qnixsynapse · 2025-06-14T14:34:43Z

Yep. :)

Please merge this PR first so that I can adjust the existing kernels for split up and gate. :)

I will deduplicate the SYCL code then.

CISC · 2025-06-14T14:42:45Z

Please merge this PR first so that I can adjust the existing kernels for split up and gate. :)

The plan is to merge #14181 into this one once @ggerganov signs off on it, then backends can be updated, and once all tests go green, merge into master.

CISC · 2025-06-14T15:03:23Z

@qnixsynapse If you want you can bring the other branch up-to-date and add your changes there.

CISC · 2025-06-16T09:00:35Z

@ggerganov So, I think we are only missing the Metal swapped implementation here, and split in #14181.

At this point I think it makes sense to add those separately, then merge this first, then the other PR separately?

sgeor255 · 2025-06-17T16:38:14Z

ggml/src/ggml-sycl/element_wise.cpp

@@ -649,6 +680,33 @@ static void clamp_sycl(const T *x, T *dst, const float min,
        });
 }

+template<typename T>
+static void geglu_sycl(const T * x, const T * g, T * dst, const uint64_t k, const uint64_t n, const uint64_t o, queue_ptr main_stream) {


Nit: For geglu_sycl, reglu_sycl & swiglu_sycl the code is identical and I think you can have a single function where the gated op is passed as a template parameter.

I believe this is exactly what was done in #14181 :)

ggerganov · 2025-06-17T18:54:03Z

The change became much bigger than I initially anticipated and I'm having some second thoughts about this. There was a previous discussion about fusing ops: see #5413 (comment). I am wondering if we should directly implement the approach that @slaren proposed there.

CISC · 2025-06-17T19:36:46Z

The change became much bigger than I initially anticipated and I'm having some second thoughts about this. There was a previous discussion about fusing ops: see #5413 (comment). I am wondering if we should directly implement the approach that @slaren proposed there.

That sounds like a very good idea, should be possible to refactor this PR without too many changes to backend, I'll have a look!

jeffbolznv · 2025-06-17T19:52:55Z

Is there a plan for how to handle changes in memory allocation due to fusion? That design feels a bit incomplete. Maybe we should discuss more before you start implementing.

jeffbolznv · 2025-06-17T20:03:01Z

A very rough suggestion: no new "fused" op. Have a "use count" in the tensor that is incremented during build, so that the backend knows a tensor is only used by one op. Add a function for the backend to report that a tensor doesn't need to be written to memory while the graph is being built.

CISC · 2025-06-17T20:19:25Z

Is there a plan for how to handle changes in memory allocation due to fusion? That design feels a bit incomplete. Maybe we should discuss more before you start implementing.

Yeah, I can see there are quite a few implementation details that need to be laid out clearly here.

First though I think it would make refactoring easier if we merged the other PR into this one, any objections? @ggerganov @0cc4m @qnixsynapse

0cc4m · 2025-06-17T20:24:26Z

Is there a plan for how to handle changes in memory allocation due to fusion? That design feels a bit incomplete. Maybe we should discuss more before you start implementing.

Yeah, I can see there are quite a few implementation details that need to be laid out clearly here.

First though I think it would make refactoring easier if we merged the other PR into this one, any objections? @ggerganov @0cc4m @qnixsynapse

No, please do.

The change became much bigger than I initially anticipated and I'm having some second thoughts about this. There was a previous discussion about fusing ops: see #5413 (comment). I am wondering if we should directly implement the approach that @slaren proposed there.

Why not just have a device-specific optimization run that checks for fusable ops only if the backend supports them? Or would we not be able to cache that "improved" graph for each repetition of the graph run? Then it might be too expensive.

ngxson · 2025-06-17T20:33:20Z

I'm just wondering if fusing ops in this particular case (swiglu/geglu/etc) can actually improve the performance in a significant way.

The main issue with the non-fused version prior to this PR was that we need ggml_cont, and worse, we are doing cont on tensors shape n_ff

So just out of curiosity, can we do a test without ggml_cont on CUDA @JohannesGaessler ? IIUC at least on CUDA, unary ops and ggml_mul already support non-cont tensors, so it should work out of the box.

CISC · 2025-06-17T20:40:14Z

I'm just wondering if fusing ops in this particular case (swiglu/geglu/etc) can actually improve the performance in a significant way.

You can see in the other PR that even though it's not as much we are still gaining 1-3% depending on backend and model, it's not much, but it is something, and it can be a lot more if we go all out and include mul-mat in the fuse.

CISC · 2025-06-17T20:46:00Z

A very rough suggestion: no new "fused" op. Have a "use count" in the tensor that is incremented during build, so that the backend knows a tensor is only used by one op.

~~We still need to store the op chain, how else than a GGML_OP_FUSED?~~

Edit: Hmmm, so, extending @0cc4m suggestion, we could perhaps somehow present the longest possible chain (within reason) of ops to the backend and it would compile that down to the shortest possible combination of fused+single ops and run it.

Edit2: @jeffbolznv I think I understand what you meant now, we don't even have to prepare that chain, we can simply keep queuing ops and do the fused op once it gets the longest possible one.

jeffbolznv · 2025-06-17T21:10:36Z

We still need to store the op chain, how else than a GGML_OP_FUSED?

The backend can just look ahead. Ideally it would be nice to have a use list like in LLVM, but I don't think we need that complexity for now. The backend can just look at the next few nodes.

CISC · 2025-06-17T21:27:41Z

BTW, we need a way to disable specific ops from getting fused, f.ex. when running imatrix:

llama.cpp/tools/imatrix/imatrix.cpp

Lines 83 to 84 in 745aa53

    
           if (t->op == GGML_OP_MUL_MAT_ID) return true; // collect all indirect matrix multiplications 
        
           if (t->op != GGML_OP_MUL_MAT) return false;

CISC added 6 commits June 12, 2025 17:39

implement unary REGLU/GEGLU/SWIGLU cpu ops

c717198

relax constraints

92943e7

duplicate shape of source

319c6cb

fix ggml_vec_geglu_f16

6fe7e07

special case gated ops

7e075be

implement unary REGLU/GEGLU/SWIGLU cuda ops

1acd121

CISC added the help wanted Extra attention is needed label Jun 12, 2025

CISC requested a review from ggerganov June 12, 2025 23:25

github-actions bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Jun 12, 2025

ggerganov reviewed Jun 13, 2025

View reviewed changes

ggml/include/ggml.h Outdated Show resolved Hide resolved

CISC added 2 commits June 13, 2025 09:00

tighten constraints again

5c58196

refactor into GGML_GLU_OP

f4be71e

CISC changed the title ~~ggml : implement unary REGLU/GEGLU/SWIGLU ops~~ ggml : implement REGLU/GEGLU/SWIGLU ops Jun 13, 2025

CISC requested a review from ggerganov June 13, 2025 08:23

ggerganov approved these changes Jun 13, 2025

View reviewed changes

ggerganov requested a review from JohannesGaessler June 13, 2025 08:30

JohannesGaessler reviewed Jun 13, 2025

View reviewed changes

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

ggml/src/ggml-cuda/unary.cu Show resolved Hide resolved

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

metal : add glu kernels

564861d

ggml-ci

github-actions bot added the Apple Metal https://en.wikipedia.org/wiki/Metal_(API) label Jun 13, 2025

ngxson reviewed Jun 13, 2025

View reviewed changes

CISC added 2 commits June 13, 2025 16:10

add CUDA_GLU_BLOCK_SIZE [no ci]

4b7d4dd

more constraints and use 64bit ints

d1d3f4f

ggml-ci

JohannesGaessler reviewed Jun 13, 2025

View reviewed changes

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

ngxson reviewed Jun 13, 2025

View reviewed changes

CISC mentioned this pull request Jun 14, 2025

llama: Attempt to add ModernBert #14014

Open

Vulkan: Add GLU ops and shaders

8dc1d9f

github-actions bot added the Vulkan Issues specific to the Vulkan backend label Jun 14, 2025

CISC mentioned this pull request Jun 14, 2025

ggml : implement GLU for split up/gate #14181

Open

SYCL: Implement fused kernel GEGLU, SWIGLU and REGLU for single up+gate

95e4be0

github-actions bot added the SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language label Jun 14, 2025

Rbiessy mentioned this pull request Jun 17, 2025

sycl: add usage of enqueue_functions extension #14244

Open

sgeor255 approved these changes Jun 17, 2025

View reviewed changes

ggml : implement REGLU/GEGLU/SWIGLU ops #14158

Are you sure you want to change the base?

ggml : implement REGLU/GEGLU/SWIGLU ops #14158

Conversation

CISC commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Jun 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JohannesGaessler commented Jun 13, 2025

Uh oh!

CISC commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler commented Jun 13, 2025

Uh oh!

0cc4m commented Jun 13, 2025

Uh oh!

Uh oh!

CISC commented Jun 13, 2025

Uh oh!

ngxson Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0cc4m commented Jun 14, 2025

Uh oh!

qnixsynapse commented Jun 14, 2025

Uh oh!

CISC commented Jun 14, 2025

Uh oh!

qnixsynapse commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CISC commented Jun 14, 2025

Uh oh!

qnixsynapse commented Jun 14, 2025

Uh oh!

CISC commented Jun 14, 2025

Uh oh!

CISC commented Jun 14, 2025

Uh oh!

CISC commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Jun 17, 2025

Uh oh!

CISC commented Jun 17, 2025

Uh oh!

jeffbolznv commented Jun 17, 2025

Uh oh!

jeffbolznv commented Jun 17, 2025

Uh oh!

CISC commented Jun 17, 2025

CISC commented Jun 12, 2025 •

edited

Loading

ngxson Jun 13, 2025 •

edited

Loading

qnixsynapse commented Jun 14, 2025 •

edited

Loading

ngxson commented Jun 17, 2025 •

edited

Loading

CISC commented Jun 17, 2025 •

edited

Loading