Can we run infinity with gguf quantized models ?

[Greatz08]

I would love to try to quantized gguf model through infinity but i didnt found anything related to gguf in issues, discussions, docs so asking here.

[michaelfeil]

Sorry, gguf are not working well or are common for embeddings / rerankers. Reason is that operations are mostly compute bound, and not memory bound like llm generation.

[Greatz08]

@michaelfeil can we use embedding and reranker models gguf with infinity ? If yes then, that's the only thing i want