LLaMA

[nikita322]

Hi, please add script to install LLaMA (or oLLaMA)

[ulmentflam]

I don't believe @tteck wants to support scripts that require Nvidia GPU passthrough. I have a branch with an ollama script you are welcome to use, just use it at your own risk. It does take some time to setup Nvidia passthrough on LXC, but there are multiple guides available. https://github.com/ulmentflam/Proxmox/tree/experimental

[tteck]

The repository has several iGPU passthrough scripts, none NVIDIA.

[tteck]

Eight to be exact.

Proxmox/misc/build.func

Line 594 in 53b24ef

if [[ "$APP" == "Channels" || "$APP" == "Emby" || "$APP" == "Frigate" || "$APP" == "Jellyfin" || "$APP" == "Plex" || "$APP" == "Scrypted" || "$APP" == "Tdarr" || "$APP" == "Unmanic" ]]; then

[ulmentflam]

So would you be willing to support dedicated GPU passthrough for Nvidia if there was an optional driver install script for the PVE Shell, then an update to the build step to pass the GPU through if the driver is available with a similar methodology to the above? For some context; in my Proxmox setup I don't have an integrated graphics card to pass through, so I am doing all my hardware acceleration through dedicated Nvidia cards that are passed through for hardware acceleration, transcoding, etc...

[tteck]

No, you have to modify the Proxmox host.

[ulmentflam]

Got it. Now I think about it, multiple steps in the process require you edit host files. VFIO was simpler but you can't mount that to multiple LXC for good reason. 🤔 Worth the discussion though, as I am sure you will have more interest in self-hosting LLMs and Diffusion models that require dedicated GPUs.

[havardthom]

Added option for installing Ollama in Open WebUI LXC script https://github.com/tteck/Proxmox/blob/main/CHANGELOG.md#2024-10-27

Alternatively there is also a separate Ollama LXC script which builds Ollama from source and supports Intel iGPU passthrough (very long install time): bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/ollama.sh)"