Retrieving the context length

[HarvieKrumpet]

I know this sort of defeats your minimal bindings methodology. But I am finding it crucial to know what llama.cpp set the context length to. in the situation where I set it to -1 and let it choose the context size. The reason for letting it choose is because it is becoming tedious to set these nowadays as they are all over the board. I need to know this context size for trimming the context window. All works of course if I explicitly set this. but only llama.cpp knows this value when reads the gguf's native size. I could potentially just start hacking away at the native bindings and do it myself. but then I break when you update. Possibly this falls under some very, very basic helper function category you could just include as a sort of extension.

[dranger003]

You mean something like LlmEngine.ContextLength?

[HarvieKrumpet]

yes, but I set that. the engine doesn't come along and update it after a load. We can auto-set the context by setting it to -1 or 0. which then the engine itself sets it to whatever the gguf says it is. I could potentially hack in and get the value via the native bindings. But then I break abstraction you have provided. it would be great if you come along and update all the values you have learned during the model load. adding all the extra stuff also beyond just context. Right now I have hacked it in, but to do so, I have to include the usings of the native bindings which means I have now created a unnecessary dependency

[HarvieKrumpet]

same goes actually for the layers, if we set it above my actual cuda layers. the engine will trim it back. but I don't get to know that value without calling llama directly. In days past, all of this was critical to set manually. but nowadays things are easier without digging around model cards trying to determine these values. by setting context to -1 and layers to 99 it will work assuming you have the memory for it.

[dranger003]

I added new properties just now, I think this is what you were looking for.

[HarvieKrumpet]

worked like a charm! thank you for dealing with this so fast. So far I have been able to load models with your lib that usually would have given me an out of memory error, from other middleware libs. I think it's due to them bulking out their libraries with features and attempting to take over the context flow management which none of them do very well.

One last issue, getting the token count just before I issue the prompt command. Which is totally necessary to work over my context flow "before" it hits your inference loop, of which you test for it and then bail out of the inference.

The proper way is to never feed your inference loop a context that will knowingly will crash out.

But this becomes complicated because your expecting a llmMessages array.
This means I need to loop over my entire contextflow myself to build a string. and feed that into native tokenize just to get the tokens. THEN you will do the same type of loop to tokenize so you can pass it to begin inference. This I am sure can't be an efficient use of the llm. And once again I have touched the native llm bindings in an area that should be abstracted away from me.

What is needed is a two part process. where your still fed the llmMessages for tokenization but return to me with a structure I can pass back to you for step 2, this gives me a moment to read the token count from the span, so I can use a sliding context window technique and redo the messages to re-issue the pass 1 again knowing it will fit next time around.

This structure will have everything you prepared to begin inference. And no duplication of tokenization effort (if my context flow fits)

Then in pass 2 you actually perform the inference stream async loop as usual and all is well in a efficient manor.

Currently the lib will work until we hit the max context length and just crash out. And must run tokenize twice on every prompt once for me, and once for your loop. Which your loop was carefully crafted and I have seen your cleverness in there. :)

[dranger003]

Fair point, I added a token count at the prompt input just now, so that should help you. As you have noted, the library doesn't perform context management, I think this is a task for the implementer. There are differents ways to manage context, but also as you pointed out, none of them I know are truely effective and it gets even messier if you need to share the context across multiple clients.

[HarvieKrumpet]

thank you, I implemented it. I can finally stop patching into your library :)

Just a small point though. there is a duplication of work. which we both must tokenize. this may seem trivial at first. until we start hitting contexts over a million or more (in the future) and/or utilize multiple users and multiple models all hammering away on the llmengine filling that queue up concurrently. we will be wishing to reduce some of the workload very quickly here.

As you also process the tokens again later in your worker thread (concurrently). but if on my first tokenize pass, I keep the list and pass you the actual token list. You can eliminate the need of even needing the llmMessage during the prompt phase. the first tokenize phase will have already built in all the bos and eos into the list so you don't even need to know the roles anymore. and that also eliminates the need for the llmmessage based list.
so if you had a 2nd version of prompt that took in a token list instead of the llmmessages. we could run tokenize once. I can use it as it is and it will work. and I suppose the token list is smaller than the text based. another efficiency win.
And this also means llmMessage would not be integral to the prompt system and can be relegated to being an structure for the examples. another win.

[dranger003]

Yes, I thought of this early on and rather decided to just use a List<LlmMessage>. The tokenization is just a KV lookup so it is very fast, even for 100K+ tokens. Separately keeping track of tokens in parallel to the KV cache isn't ideal.

[AshD]

This is more of a question - How do you set the number for GpuLayers for the most performance possible.

I have a Windows client app, so the GPULayers has to be dynamic based on the user's GPU memory and number of layers and Model size of the GGUF model.

[dranger003]

There isn't a perfect answer to this, as far as I know this is trial and error.

One important thing I found out about is that you need to set your NVIDIA driver to not fallback on shared memory when running out of memory, because things become extremely slow then.

So with the driver setting to not allow a fallback, then it's about trying a number of layers until it runs. I sometimes take that final number of layers and subtract one as a buffer, as I have noticed with longer context sometimes it eats up a little bit more memory.

[AshD]

There isn't a perfect answer to this, as far as I know this is trial and error.

One important thing I found out about is that you need to set your NVIDIA driver to not fallback on shared memory when running out of memory, because things become extremely slow then.

So with the driver setting to not allow a fallback, then it's about trying a number of layers until it runs. I sometimes take that final number of layers and subtract one as a buffer, as I have noticed with longer context sometimes it eats up a little bit more memory.

That's what I do when I run the models locally on my computer. Trial and Error.

Looking for better algorithmic solution for my Windows client app. One idea was to create a database of settings for number of parameters, GPU memory sizes and number of layers. It would have been nice if llama.cpp figured this out dynamically when loading the model.

[dranger003]

The database idea is neat given a set of tested parameters. I know there has been a few attempts to add this support in llama.cpp, but unfortunately this isn't as trivial as we think - you can take a look at this PR for more details: ggml-org/llama.cpp#6502