NoneType Error when using TransformersEmbeddings

[Kailuo-Lai]

When I try to use "sentence-transformers/all-MiniLM-L12-v2" embedding model downloaded from huggingface, I encounter a wired problem

Code

from bigdl.llm.langchain.embeddings import TransformersEmbeddings
embeddings = TransformersEmbeddings.from_model_id(model_id="./checkpoints/all-MiniLM-L12-v2")

Output

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 2
      1 llm = TransformersLLM.from_model_id_low_bit("./checkpoints/Llama-2-7b-chat-hf-INT4/")
----> 2 embeddings = TransformersEmbeddings.from_model_id(model_id="./checkpoints/all-MiniLM-L12-v2")

File ~/anaconda3/envs/llm-tutorial-new/lib/python3.9/site-packages/bigdl/llm/langchain/embeddings/transformersembeddings.py:117, in TransformersEmbeddings.from_model_id(cls, model_id, model_kwargs, **kwargs)
    114 except:
    115     tokenizer = LlamaTokenizer.from_pretrained(model_id, **_model_kwargs)
--> 117 model = AutoModel.from_pretrained(model_id, load_in_4bit=True, **_model_kwargs)
    119 if "trust_remote_code" in _model_kwargs:
    120     _model_kwargs = {
    121         k: v for k, v in _model_kwargs.items() if k != "trust_remote_code"
    122     }

File ~/anaconda3/envs/llm-tutorial-new/lib/python3.9/site-packages/bigdl/llm/transformers/model.py:95, in _BaseAutoModelClass.from_pretrained(cls, *args, **kwargs)
     93         kwargs["pretraining_tp"] = 1
     94     q_k = load_in_low_bit if load_in_low_bit else "sym_int4"
---> 95     model = cls.load_convert(q_k, optimize_model, *args, **kwargs)
     96 else:
     97     # load default
     98     model = cls.HF_Model.from_pretrained(*args, **kwargs)

File ~/anaconda3/envs/llm-tutorial-new/lib/python3.9/site-packages/bigdl/llm/transformers/model.py:125, in _BaseAutoModelClass.load_convert(cls, q_k, optimize_model, *args, **kwargs)
    123     model.config.update({"bigdl_lcmu_enabled": False})
...
    185     if hasattr(model.config, "padded_vocab_size") and model.config.padded_vocab_size == 65024:
    186         # chatglm2-6b
    187         modeling_module_name = model.__class__.__module__

TypeError: 'NoneType' object is not subscriptable

Environment

• bigdl-llm: 2.4.0b20231014

[Kailuo-Lai]

I think I find the root of this problem.
When I change "bigdl/lm/langchain/embeddings/transformersembeddings" line 117 from

model = AutoModel.from_pretrained(model_id, load_in_4bit=True, **_model_kwargs)

into

model = AutoModel.from_pretrained(model_id, load_in_4bit=False, **_model_kwargs)

I solve this problem. I think may be "sentence-transformers/all-MiniLM-L12-v2" doesn't support low-bit optimization. Can someone add this argument to function "from_model_id" or maybe there are other methods to solve the error.

[rnwang04]

Hi @Kailuo-Lai, this problem is caused by an internal bug, and I have fixed it, and "sentence-transformers/all-MiniLM-L12-v2" can be accelrated by low-bit optimization.
You can try this code again tomorrow (so that new bigdl-llm package is released).

[Kailuo-Lai]

Hi @Kailuo-Lai, this problem is caused by an internal bug, and I have fixed it, and "sentence-transformers/all-MiniLM-L12-v2" can be accelrated by low-bit optimization. You can try this code again tomorrow (so that new bigdl-llm package is released).

Thank you, I will try it later.