Can I use llama.cpp with ragflow

[lll123github]

Can I use llama.cpp with ragflow (not Ollama)
I want to apply some models on the Raspberry Pi 4B with ImmortalWrt (something like OpenWrt) , using llama.cpp to fully use the capability of computation. But I see the support for llama.cpp is not shown?

[dosubot[bot]]

RAGFlow does not natively support llama.cpp as a direct backend. Instead, it integrates with LLMs via HTTP APIs, such as those provided by Ollama, LocalAI, or other OpenAI-compatible servers. There is no built-in class, provider, or documented workflow for connecting RAGFlow directly to a standalone llama.cpp instance without an API wrapper. All official documentation and code point to using an HTTP server as the bridge—typically Ollama or LocalAI—which can themselves use llama.cpp under the hood for inference on devices like the Raspberry Pi 4B.

If you want to use llama.cpp on your Pi, the recommended approach is to run a compatible API server (like LocalAI) that wraps llama.cpp, then configure RAGFlow to connect to that server using the "LocalAI" provider and the appropriate base URL. Direct integration with llama.cpp binaries or libraries is not currently supported in RAGFlow’s codebase or configuration system. For more details, see the discussion and code references:

• Supported providers and integration pattern
• Embedding model backend support
• Official documentation on local model deployment

If your use case requires direct llama.cpp integration, it would require custom development to add a new provider and handler in RAGFlow. For now, using LocalAI or a similar API wrapper is the practical solution.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}