Converting GGML->GGUF: ValueError: Only GGJTv3 supported

[jboero]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [X ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ X] I carefully followed the README.md.
• [X ] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [X ] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

My GGML converted models should be easy to convert to GGUF.
I know the conversion tools aren't guaranteed but I'd like to file this one in case anybody else has a workaround or more version flexible option. I would love to see any version of GGML/GGJT supported if possible. Instead my GGML files converted earlier are apparently not supported for conversion to GGUF.

Is there any tool to show the standard version details of a model file? Happy to contribute one if there isn't.

Current Behavior

python3 ./convert-llama-ggmlv3-to-gguf.py -i llama-2-70b/ggml-model-f32.bin -o test.gguf
=== WARNING === Be aware that this conversion script is best-effort. Use a native GGUF model if possible. === WARNING ===

* Scanning GGML input file
Traceback (most recent call last):
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 353, in <module>
    main()
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 335, in main
    offset = model.load(data, 0)
             ^^^^^^^^^^^^^^^^^^^
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 125, in load
    offset += self.validate_header(data, offset)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 121, in validate_header
    raise ValueError('Only GGJTv3 supported')
ValueError: Only GGJTv3 supported

Environment and Context

Working with models

• Physical (or virtual) hardware you are using, e.g. for Linux:
  Physical Fedora 38, probably irrelevant give the Python.

$ lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  56
  On-line CPU(s) list:   0-55
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):           2
    Stepping:            1
    CPU(s) scaling MHz:  40%
    CPU max MHz:         3200.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            3990.92
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts a
                         cpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_per
                         fmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes
                         64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_
                         2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowp
                         refetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb st
                         ibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bm
                         i2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc
                          cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi md_clear flush_l1d
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   896 KiB (28 instances)
  L1i:                   896 KiB (28 instances)
  L2:                    7 MiB (28 instances)
  L3:                    70 MiB (2 instances)
NUMA:                    
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-13,28-41
  NUMA node1 CPU(s):     14-27,42-55

• Operating System, e.g. for Linux:

$ uname -a
Linux z840 6.4.12-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 23 17:46:49 UTC 2023 x86_64 GNU/Linux

• SDK version, e.g. for Linux:

$ python3 --version
Python 3.11.4
$ make --version
$ g++ --version

Failure Information (for bugs)

python3 ./convert-llama-ggmlv3-to-gguf.py -i llama-2-70b/ggml-model-f32.bin -o test.gguf
=== WARNING === Be aware that this conversion script is best-effort. Use a native GGUF model if possible. === WARNING ===

* Scanning GGML input file
Traceback (most recent call last):
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 353, in <module>
    main()
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 335, in main
    offset = model.load(data, 0)
             ^^^^^^^^^^^^^^^^^^^
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 125, in load
    offset += self.validate_header(data, offset)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH]/llama.cpp/convert-llama-ggmlv3-to-gguf.py", line 121, in validate_header
    raise ValueError('Only GGJTv3 supported')
ValueError: Only GGJTv3 supported

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. step 1 convert any of the PTH models to GGML (using previous unversioned commits of convert)
2. step 2 convert the GGML to GGUF with the command given above.

[KerfuffleV2]

My GGML converted models should be easy to convert to GGUF.

There are versions of GGML that had really strange, difficult to support stuff like multi-part files, including individual tensors split across (or duplicated) across the files, etc. So supporting all versions of the previous GGML formats definitely isn't easy or simple.

It also looks like you're converting without the model metadata, and converting the vocabulary also isn't perfect. Even if you mess around with this, you're going to get a model that's lower quality than one that was directly converted to GGUF.

Is it really impractical for you to just download the GGUF version? edit: Also, do you know what version the file actually is?

[jboero]

If everyone needs to re-download 200GB+ every time there's a schema change I hate to think of the egress costs to the org but I guess that's the way for now. Any documentation for the header info that I might be able to write an info helper app? Thx

…

On Sun, Sep 3, 2023, 14:36 Kerfuffle ***@***.***> wrote: My GGML converted models should be easy to convert to GGUF. There are versions of GGML that had really strange, difficult to support stuff like multi-part files, including individual tensors split across (or duplicated) across the files, etc. So supporting all versions of the previous GGML formats definitely isn't easy or simple. It also looks like you're converting without the model metadata, and converting the vocabulary also isn't perfect. Even if you mess around with this, you're going to get a model that's lower quality than one that was directly converted to GGUF. Is it really impractical for you to just download the GGUF version? — Reply to this email directly, view it on GitHub <#2990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZP3DHJTTZMAG2WZFGMSVDXYSBWVANCNFSM6AAAAAA4JHU6P4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Ph0rk0z]

Is it really impractical for you to just download the GGUF version?

For those of us who use 30b/70B models yes. It very much is impractical to download 40GB over and over again. To download unquantized models it's also impractical because they are hundreds of GB. If you are downloading 10 - 20 models over time this is virtually impossible due to data caps and internet speeds.

[KerfuffleV2]

@jboero

Any documentation for the header info that I might be able to write an info

Not really, kind of have to just figure it out by looking at the loading code for different versions in the projects that have supported GGML. You didn't answer my question about what version you have. If you can load it with an older llama.cpp version I think it will say what it is when it gets loaded.

Or if you can show me the first 10 or so bytes in a hexdump. For example on Linux something like:

hexdump < /path/to/model.bin | head -2

---

@Ph0rk0z

Is it really impractical for you to just download the GGUF version?

For those of us who use 30b/70B models yes.

Just to be clear, I wasn't saying that in a snarky way. People tend to just make issues when they run into a problem, even if there's a relatively easy workaround. Since converting the GGML models isn't ideal in the first place, I was just checking to make sure there wasn't an easier way to deal with this.

[Ph0rk0z]

I don't wanna hijack but I have a similar problem now. I converted a bunch of GGML to GGUF and they worked fine. Now I directly downloaded 2 quants in GGUF and am getting huge repetition problems at long context. I also have the GPTQ version and these issues aren't present. It isn't the model.

So now, I'm stuck downloading a GGML of the same and converting it to see if that will work. But this means that I have downloaded over 120GB of the same model, not counting the GPTQ. All due to format.

At the end, I also don't know if the other models are GGUFv1 or GGUFv2 from either a direct quant or conversion scripts. GGUFv1 goes away in a month. Will they convert after? Will some other strange bug like this occur? You can see how this is frustrating, right?

[jboero]

I had that same issue with massive repetition... I thought it was just a fluke but maybe I'm not the only one. Anyway that's probably for a separate issue but I'll do my best to write up a quick version/info app as there doesn't seem to be any. Good tip on loading with an old build. John

…

On Sun, Sep 3, 2023, 17:19 Forkoz ***@***.***> wrote: I don't wanna hijack but I have a similar problem now. I converted a bunch of GGML to GGUF and they worked fine. Now I directly downloaded 2 quants in GGUF and am getting huge repetition problems at long context. I also have the GPTQ version and these issues aren't present. It isn't the model. So now, I'm stuck downloading a GGML of the same and converting it to see if that will work. But this means that I have downloaded over 120GB of the same model, not counting the GPTQ. All due to format. At the end, I also don't know if the other models are GGUFv1 or GGUFv2 from either a direct quant or conversion scripts. GGUFv1 goes away in a month. Will they convert after? Will some other strange bug like this occur? You can see how this is frustrating, right? — Reply to this email directly, view it on GitHub <#2990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZP3DEFNP2HJKG5O4LVTOTXYSUXZANCNFSM6AAAAAA4JHU6P4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[KerfuffleV2]

@Ph0rk0z

I converted a bunch of GGML to GGUF and they worked fine.

Did you convert using metadata? (The --model-metadata-option for convert-llama-ggmlv3-to-gguf.py)?

If not, that might be something to test: see if doing the GGML conversion with metadata leads to the repetition problem you mentioned.

GGUFv1 goes away in a month. Will they convert after?

I got your back with #2931. You can now use the quantize tool in copy mode to repackage your GGUFv1 files to GGUFv2.

---

@jboero

Anyway that's probably for a separate issue but I'll do my best to write up a quick version/info app as there doesn't seem to be any.

The file will start with:

1. bytes lmgg (ggml reversed). This is the old/original GGML version.
2. bytes fmgg followed by a little endian u32 for version. That's GGMF.
3. bytes tjgg follow by a little endian u32 for version. That's GGJT.

That's why I was asking you for a hexdump of the beginning of the file. (By the way, I'm the person who wrote the convert-llama-ggmlv3-to-gguf.py tool.)

I think the main difference in GGMF from GGML was that the version was added and vocabulary items added score (a f32 with each vocab entry). GGJT added padding/alignment. I don't remember what's different between the different GGJT versions. Just to be clear, this is just talking about differences in the structure of the file not really the content. I'm pretty sure there were some changes to quantization formats in there, and something like that is pretty much impossible to convert.

You can possibly try changing:

if bytes(data[offset:offset + 4]) != b'tjgg' or struct.unpack('<I', data[offset + 4:offset + 8])[0] != 3:

to

if bytes(data[offset:offset + 4]) != b'tjgg':

(should be around line 120 in the script)

This will just completely ignore the version (still only supports GGJT) and blindly attempt to proceed. It may or may not actually work.

[Ph0rk0z]

I didn't use metadata. I only used the script. I'll also try the repeating models in GGUFv2 because why not. I won't have a GGML copy of the model to attempt to convert until tomorrow.

[KerfuffleV2]

I'll also try the repeating models in GGUFv2 because why not.

You shouldn't notice any difference. The only thing GGUFv2 did (as far as I know) is change some types to 64bit to allow expressing larger values. Existing files wouldn't have any values over 32bit so there wouldn't be a visible difference.

I didn't use metadata. I only used the script.

Alright. Well, if you notice that converting with metadata leads to the repetition issue but converting using only the GGML file doesn't that would mean it's something with the new vocab stuff. GGUF is supposed to be better in that regard though and converting GGML to GGUF without the metadata is imperfect. So it's likely to be worse than the original GGML file in terms of quality.

[danielbrdz]

Does anyone know how to solve the problem:
ValueError: Only GGJTv3 supported

I had already successfully converted GGML to GGUF last week, but I updated the llama.cpp and now I get this error.

[KerfuffleV2]

I had already successfully converted GGML to GGUF last week

What? The GGML to GGUF conversion script has only ever supported GGJTv3. Maybe you successfully converted a GGJTv3 file and then tried to convert a GGML file of a different version (non GGJTv3). As for possibly ways to deal with that, please read through the other posts in this issue.

I can't help you if I don't know the version of the GGML file you're trying to convert.

[jboero]

Added a magic file: #3011

…

On Mon, Sep 4, 2023 at 2:27 AM Kerfuffle ***@***.***> wrote: I had already successfully converted GGML to GGUF last week What? The GGML to GGUF conversion script has only ever supported GGJTv3. Maybe you successfully converted a GGJTv3 file and then tried to convert a GGML file of a different version (non GGJTv3). As for possibly ways to deal with that, please read through the other posts in this issue. I can't help you if I don't know the version of the GGML file you're trying to convert. — Reply to this email directly, view it on GitHub <#2990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZP3DAWYQLNBCSI73JD4CTXYUU7NANCNFSM6AAAAAA4JHU6P4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Ph0rk0z]

Got the K5M GGML. Running it as GGML it still does some repetition. I also converted it to GGUF (v1/v2 makes no difference). The extreme repetition where it generates the same thing over and over happens about 1000 tokens later in the converted model. It can now be broken via mirostat with high TAU. You'll still get your bits of the previous messages but at least now the plot moves forward. Every message may start with the robot smiles slyly but the rest of the contents will be different.

Something is definitely wrong with how this was quantized. The GPTQ model had none of these problems. I will try to run it with pure exllama so that there are less samplers for a final check. I'm not sure what else to do or why it would be such a big difference between the two formats or why llama.cpp performs so poorly in this department. So far none of my other models have suffered from this and GGUF/GGML have been smarter than their GPTQ equivalents.

Model is: https://huggingface.co/nRuaif/fiction.live-Kimiko-V2-70B and I'm using the bloke's quants.