Local Model

[aasawa20]

I have a local model that I want to use with infinity because the environment I am working with does not permit downloads from Hugging Face. How can I point infinity to my local model instead of it downloading the model from hugging face? I have looked at other discussions around this issue but they don't seem to help.

[michaelfeil]

Hey @aasawa20

• there is a short tutorial in the https://michaelfeil.eu/infinity/0.0.39/deploy/ on how to bake it into your docker image.
• there is a --preload-only flag infinity_emb v2 --preload-only
• you can do the above in a github ci and push to docker hub.

Hope these are some helpful pointers.
If you are not allowed to use hf.co at all (even to bake it into the docker image, or pre-populate cache) my honest answer I cannot help you. You would need to download a lot and place in the right besides the sentence_transformers_config.json, tokenizer.json, model.safetensors of your $HF_HOME dir. I would not recommend going down this rabbit hole.

[aasawa20]

Thanks for this. I actually got it to work by setting HF_HUB_OFFLINE=1 and making sure the model folder is in my working directory.