local-llm-stack

Small Linux-focused scripts for running Open WebUI with a local llama.cpp server backend.

The project is intentionally simple: one wrapper command, a few lifecycle scripts, one runtime data directory, and one models.ini file. It is built for AMD-style Linux setups and supports the official llama.cpp Vulkan and ROCm server images.

What It Does

• Starts Open WebUI.
• Starts a llama.cpp Vulkan or ROCm backend.
• Keeps large runtime data outside the repository.
• Shows stack health, model state, and storage usage.
• Provides a safe interactive uninstall flow.

Requirements

• Linux
• Bash
• Docker
• curl
• python3
• standard Linux userland tools such as mktemp, cp, mkdir, du, df, grep, and stat
• GPU device access:
  • Vulkan: /dev/dri/renderD*
  • ROCm: /dev/kfd and /dev/dri/renderD*

If your user is not in the Docker group, setup image pulls, start, stop, logs, update, and uninstall can ask for your sudo password when needed. Status and storage remain non-interactive.

To use Docker without sudo, run sudo usermod -aG docker "$USER", then log out and back in.

Do not run the wrapper itself with sudo; that can make config and data paths point at /root. Run ./llm-stack ... as your normal user and let Docker operations ask for sudo when needed.

Port checks use ss when available and fall back to /proc/net/tcp*.

./llm-stack setup checks required commands up front and lists anything missing before it writes configuration. It also rejects ports that are already in use and asks you to stop known stack containers before setup or reconfiguration.

Quick Start

git clone https://github.com/MSC2016/local-llm-stack
cd local-llm-stack

./llm-stack setup
./llm-stack start vulkan

Open WebUI:

http://localhost:3000

llama.cpp OpenAI-compatible API:

http://localhost:8080/v1

Commands

./llm-stack setup
./llm-stack start vulkan
./llm-stack start rocm
./llm-stack start webui
./llm-stack status
./llm-stack storage
./llm-stack logs
./llm-stack logs webui
./llm-stack stop
./llm-stack update
./llm-stack uninstall

Setup

./llm-stack setup

Creates local configuration and runtime directories. By default, runtime data goes here:

$HOME/.local/share/local-llm-stack

Useful options:

./llm-stack setup --defaults
./llm-stack setup --reconfigure
./llm-stack setup --pull-images
./llm-stack setup --no-pull-images

Start

./llm-stack start vulkan
./llm-stack start rocm
./llm-stack start webui

Starting a backend also starts Open WebUI unless START_WEBUI_WITH_BACKEND=false is set in config.

The backend scripts check for common problems before starting: missing GPU devices, an already-running backend, stale container names, identical service ports, and occupied ports. Starting a backend that is already running exits successfully without creating another container.

Status

./llm-stack status
./llm-stack status --verbose

status is read-only. It shows the current stack state, service health, grouped model presets, dependency checks, Docker access, GPU devices, runtime config, image presence when Docker inspection is available, and storage summary without creating runtime directories or copying config files.

It exits with status 1 when the backend or Open WebUI health check fails, after printing the complete report.

Port state is shown once in the main status summary to avoid duplicate output. A listening WebUI/backend port is expected and shown with a green check when the stack is running.

If the HTTP health checks pass but Docker inspection is unavailable, the stack may still be running correctly. status and storage do not ask for sudo because diagnostics should not block on a password prompt. Docker-changing commands ask when they need access, or you can add your user to the Docker group. Avoid sudo ./llm-stack status; that runs the whole wrapper as root and can make config/data paths point at /root.

Status output uses icons and ANSI color.

Model states are displayed in friendlier terms:

• not loaded: configured, but not currently using memory.
• downloading/preparing: the backend is fetching or preparing the model.
• loading from cache: cached locally and being loaded into memory.
• cached, idle: downloaded locally, but currently sleeping to save resources.

Synthetic backend aliases such as default are hidden unless you explicitly define them in models.ini.

Cache-only backend entries are hidden by default to keep the preset list focused. Use ./llm-stack status --show-cache-models if you want to see those raw llama.cpp cache entries.

Loading, loaded, running, or unknown active models are highlighted in red. Cached idle and unloaded models remain unhighlighted. Single models show their state directly in the header. Groups with named variants get a [current: ...] badge, and the selected variant is marked with *.

When something needs attention, status prints a short Debug section with the next useful commands.

Storage

./llm-stack storage

Shows important paths, disk usage, free space, and whether the Open WebUI Docker volume exists. This command is read-only.

Model cache deletion is intentionally not automated yet. The llama.cpp/Hugging Face cache layout can be non-obvious, and this project avoids destructive cleanup unless it can do it reliably.

Logs

./llm-stack logs
./llm-stack logs webui
./llm-stack logs vulkan
./llm-stack logs rocm

./llm-stack logs follows the currently running backend logs.

Stop

./llm-stack stop

Stops and removes the llama.cpp backend and Open WebUI containers. Persistent Open WebUI data remains in its Docker volume.

Update Images

./llm-stack update

Pulls the configured Docker images.

Restart the stack afterward to run the updated images.

Uninstall

./llm-stack uninstall

Runs an interactive uninstall. The repository directory is never removed.

You choose whether to remove:

• llama.cpp backend containers
• Open WebUI container
• Open WebUI Docker volume and chat data
• local preferences in config/config.env
• LLM_ROOT data, including model cache and runtime config
• Docker images used by the stack

Preview the flow without deleting anything:

./llm-stack uninstall --dry-run

For a full uninstall, answer yes to all prompts. To keep preferences, keep config/config.env. To keep downloaded models/cache, keep LLM_ROOT.

Configuration

Default configuration is committed here:

config/config.env.default

User configuration is generated here and ignored by Git:

config/config.env

Important values:

LLM_ROOT="$HOME/.local/share/local-llm-stack"
BACKEND_PORT="8080"
WEBUI_PORT="3000"
DEFAULT_BACKEND="vulkan"
MODELS_MAX="4"

LLM_ROOT must be an absolute data directory and cannot be /, your home directory, or the repository itself.

Runtime data:

$LLM_ROOT/
├── cache/
├── models/
└── config/
    └── models.ini

Models

The repository ships with:

config/models.ini

During setup, it is copied to:

$LLM_ROOT/config/models.ini

That runtime file is what the llama.cpp container uses. Edit it to add, remove, or tune presets:

nano "$LLM_ROOT/config/models.ini"

Models are not pre-downloaded. llama.cpp downloads Hugging Face GGUF files lazily when a preset is loaded. A [default] preset is not required; add one only if you want to expose a model under that exact name.

Open WebUI Connection

The Open WebUI container is created with:

OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1
OPENAI_API_KEY=local

If models do not appear in Open WebUI, configure the connection manually in Open WebUI admin settings:

Base URL: http://host.docker.internal:8080/v1
API key: local

Storage Notes

For best reliability and performance, prefer ext4 or btrfs for LLM_ROOT.

NTFS-3G can work for large GGUF files, but it is not ideal for active cache or many small files. Avoid the Linux kernel ntfs3 driver for this workload; it has caused freezes or I/O hangs in testing.

Safety Notes

• Do not expose port 8080 to the public internet.
• Open WebUI's first account may become the admin account.
• config/config.env is sourced by Bash. Do not paste untrusted content into it.
• Destructive cleanup is only done by ./llm-stack uninstall, and it asks before removing data.

Project Layout

llm-stack           # main wrapper command
scripts/            # user-facing command implementations
lib/                # shared shell helpers
config/             # default env and model presets

Status code is split by responsibility:

lib/status.sh       # status formatting and health checks
lib/models.sh       # llama.cpp model status formatting
lib/storage.sh      # storage and path reporting

Development Checks

Run the syntax check before committing:

bash -n llm-stack lib/*.sh scripts/*.sh

Install and run the same shell checks used by CI:

sudo apt-get install shellcheck shfmt
bash -n llm-stack lib/*.sh scripts/*.sh
shellcheck -x -P SCRIPTDIR llm-stack lib/*.sh scripts/*.sh
shfmt -i 2 -d llm-stack lib/*.sh scripts/*.sh