Skip to content

Commit 2d64269

Browse files
mudlermudler
authored
feat: Add backend gallery (#5607)
* feat: Add backend gallery This PR add support to manage backends as similar to models. There is now available a backend gallery which can be used to install and remove extra backends. The backend gallery can be configured similarly as a model gallery, and API calls allows to install and remove new backends in runtime, and as well during the startup phase of LocalAI. Signed-off-by: Ettore Di Giacinto <[email protected]> * Add backends docs Signed-off-by: Ettore Di Giacinto <[email protected]> * wip: Backend Dockerfile for python backends Signed-off-by: Ettore Di Giacinto <[email protected]> * feat: drop extras images, build python backends separately Signed-off-by: Ettore Di Giacinto <[email protected]> * fixup on all backends Signed-off-by: Ettore Di Giacinto <[email protected]> * test CI Signed-off-by: Ettore Di Giacinto <[email protected]> * Tweaks Signed-off-by: Ettore Di Giacinto <[email protected]> * Drop old backends leftovers Signed-off-by: Ettore Di Giacinto <[email protected]> * Fixup CI Signed-off-by: Ettore Di Giacinto <[email protected]> * Move dockerfile upper Signed-off-by: Ettore Di Giacinto <[email protected]> * Fix proto Signed-off-by: Ettore Di Giacinto <[email protected]> * Feature dropped for consistency - we prefer model galleries Signed-off-by: Ettore Di Giacinto <[email protected]> * Add missing packages in the build image Signed-off-by: Ettore Di Giacinto <[email protected]> * exllama is ponly available on cublas Signed-off-by: Ettore Di Giacinto <[email protected]> * pin torch on chatterbox Signed-off-by: Ettore Di Giacinto <[email protected]> * Fixups to index Signed-off-by: Ettore Di Giacinto <[email protected]> * CI Signed-off-by: Ettore Di Giacinto <[email protected]> * Debug CI * Install accellerators deps Signed-off-by: Ettore Di Giacinto <[email protected]> * Add target arch * Add cuda minor version Signed-off-by: Ettore Di Giacinto <[email protected]> * Use self-hosted runners Signed-off-by: Ettore Di Giacinto <[email protected]> * ci: use quay for test images Signed-off-by: Ettore Di Giacinto <[email protected]> * fixups for vllm and chatterbox Signed-off-by: Ettore Di Giacinto <[email protected]> * Small fixups on CI Signed-off-by: Ettore Di Giacinto <[email protected]> * chatterbox is only available for nvidia Signed-off-by: Ettore Di Giacinto <[email protected]> * Simplify CI builds Signed-off-by: Ettore Di Giacinto <[email protected]> * Adapt test, use qwen3 Signed-off-by: Ettore Di Giacinto <[email protected]> * chore(model gallery): add jina-reranker-v1-tiny-en-gguf Signed-off-by: Ettore Di Giacinto <[email protected]> * fix(gguf-parser): recover from potential panics that can happen while reading ggufs with gguf-parser Signed-off-by: Ettore Di Giacinto <[email protected]> * Use reranker from llama.cpp in AIO images Signed-off-by: Ettore Di Giacinto <[email protected]> * Limit concurrent jobs Signed-off-by: Ettore Di Giacinto <[email protected]> --------- Signed-off-by: Ettore Di Giacinto <[email protected]> Signed-off-by: Ettore Di Giacinto <[email protected]>
1 parent a7a6020 commit 2d64269

File tree

114 files changed

+3979
-1365
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

114 files changed

+3979
-1365
lines changed

‎.github/dependabot.yml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,6 @@ updates:
6161
directory: "/backend/python/openvoice"
6262
schedule:
6363
interval: "weekly"
64-
- package-ecosystem: "pip"
65-
directory: "/backend/python/parler-tts"
66-
schedule:
67-
interval: "weekly"
6864
- package-ecosystem: "pip"
6965
directory: "/backend/python/rerankers"
7066
schedule:

‎.github/workflows/image-pr.yml

Lines changed: 4 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,12 @@ concurrency:
99
cancel-in-progress: true
1010

1111
jobs:
12-
extras-image-build:
12+
image-build:
1313
uses: ./.github/workflows/image_build.yml
1414
with:
1515
tag-latest: ${{ matrix.tag-latest }}
1616
tag-suffix: ${{ matrix.tag-suffix }}
1717
ffmpeg: ${{ matrix.ffmpeg }}
18-
image-type: ${{ matrix.image-type }}
1918
build-type: ${{ matrix.build-type }}
2019
cuda-major-version: ${{ matrix.cuda-major-version }}
2120
cuda-minor-version: ${{ matrix.cuda-minor-version }}
@@ -36,36 +35,24 @@ jobs:
3635
fail-fast: false
3736
matrix:
3837
include:
39-
# This is basically covered by the AIO test
40-
# - build-type: ''
41-
# platforms: 'linux/amd64'
42-
# tag-latest: 'false'
43-
# tag-suffix: '-ffmpeg'
44-
# ffmpeg: 'true'
45-
# image-type: 'extras'
46-
# runs-on: 'arc-runner-set'
47-
# base-image: "ubuntu:22.04"
48-
# makeflags: "--jobs=3 --output-sync=target"
4938
- build-type: 'cublas'
5039
cuda-major-version: "12"
5140
cuda-minor-version: "0"
5241
platforms: 'linux/amd64'
5342
tag-latest: 'false'
5443
tag-suffix: '-cublas-cuda12-ffmpeg'
5544
ffmpeg: 'true'
56-
image-type: 'extras'
57-
runs-on: 'arc-runner-set'
45+
runs-on: 'ubuntu-latest'
5846
base-image: "ubuntu:22.04"
5947
makeflags: "--jobs=3 --output-sync=target"
6048
- build-type: 'hipblas'
6149
platforms: 'linux/amd64'
6250
tag-latest: 'false'
6351
tag-suffix: '-hipblas'
6452
ffmpeg: 'false'
65-
image-type: 'extras'
6653
base-image: "rocm/dev-ubuntu-22.04:6.1"
6754
grpc-base-image: "ubuntu:22.04"
68-
runs-on: 'arc-runner-set'
55+
runs-on: 'ubuntu-latest'
6956
makeflags: "--jobs=3 --output-sync=target"
7057
- build-type: 'sycl_f16'
7158
platforms: 'linux/amd64'
@@ -74,77 +61,13 @@ jobs:
7461
grpc-base-image: "ubuntu:22.04"
7562
tag-suffix: 'sycl-f16-ffmpeg'
7663
ffmpeg: 'true'
77-
image-type: 'extras'
78-
runs-on: 'arc-runner-set'
64+
runs-on: 'ubuntu-latest'
7965
makeflags: "--jobs=3 --output-sync=target"
8066
- build-type: 'vulkan'
8167
platforms: 'linux/amd64'
8268
tag-latest: 'false'
8369
tag-suffix: '-vulkan-ffmpeg-core'
8470
ffmpeg: 'true'
85-
image-type: 'core'
8671
runs-on: 'ubuntu-latest'
8772
base-image: "ubuntu:22.04"
8873
makeflags: "--jobs=4 --output-sync=target"
89-
# core-image-build:
90-
# uses: ./.github/workflows/image_build.yml
91-
# with:
92-
# tag-latest: ${{ matrix.tag-latest }}
93-
# tag-suffix: ${{ matrix.tag-suffix }}
94-
# ffmpeg: ${{ matrix.ffmpeg }}
95-
# image-type: ${{ matrix.image-type }}
96-
# build-type: ${{ matrix.build-type }}
97-
# cuda-major-version: ${{ matrix.cuda-major-version }}
98-
# cuda-minor-version: ${{ matrix.cuda-minor-version }}
99-
# platforms: ${{ matrix.platforms }}
100-
# runs-on: ${{ matrix.runs-on }}
101-
# base-image: ${{ matrix.base-image }}
102-
# grpc-base-image: ${{ matrix.grpc-base-image }}
103-
# makeflags: ${{ matrix.makeflags }}
104-
# secrets:
105-
# dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
106-
# dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
107-
# quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
108-
# quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
109-
# strategy:
110-
# matrix:
111-
# include:
112-
# - build-type: ''
113-
# platforms: 'linux/amd64'
114-
# tag-latest: 'false'
115-
# tag-suffix: '-ffmpeg-core'
116-
# ffmpeg: 'true'
117-
# image-type: 'core'
118-
# runs-on: 'ubuntu-latest'
119-
# base-image: "ubuntu:22.04"
120-
# makeflags: "--jobs=4 --output-sync=target"
121-
# - build-type: 'sycl_f16'
122-
# platforms: 'linux/amd64'
123-
# tag-latest: 'false'
124-
# base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
125-
# grpc-base-image: "ubuntu:22.04"
126-
# tag-suffix: 'sycl-f16-ffmpeg-core'
127-
# ffmpeg: 'true'
128-
# image-type: 'core'
129-
# runs-on: 'arc-runner-set'
130-
# makeflags: "--jobs=3 --output-sync=target"
131-
# - build-type: 'cublas'
132-
# cuda-major-version: "12"
133-
# cuda-minor-version: "0"
134-
# platforms: 'linux/amd64'
135-
# tag-latest: 'false'
136-
# tag-suffix: '-cublas-cuda12-ffmpeg-core'
137-
# ffmpeg: 'true'
138-
# image-type: 'core'
139-
# runs-on: 'ubuntu-latest'
140-
# base-image: "ubuntu:22.04"
141-
# makeflags: "--jobs=4 --output-sync=target"
142-
# - build-type: 'vulkan'
143-
# platforms: 'linux/amd64'
144-
# tag-latest: 'false'
145-
# tag-suffix: '-vulkan-ffmpeg-core'
146-
# ffmpeg: 'true'
147-
# image-type: 'core'
148-
# runs-on: 'ubuntu-latest'
149-
# base-image: "ubuntu:22.04"
150-
# makeflags: "--jobs=4 --output-sync=target"

‎.github/workflows/image.yml

Lines changed: 36 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -37,24 +37,9 @@ jobs:
3737
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
3838
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
3939
strategy:
40-
# Pushing with all jobs in parallel
41-
# eats the bandwidth of all the nodes
4240
max-parallel: 2
4341
matrix:
4442
include:
45-
- build-type: 'hipblas'
46-
platforms: 'linux/amd64'
47-
tag-latest: 'auto'
48-
tag-suffix: '-hipblas-extras'
49-
ffmpeg: 'true'
50-
image-type: 'extras'
51-
aio: "-aio-gpu-hipblas"
52-
base-image: "rocm/dev-ubuntu-22.04:6.1"
53-
grpc-base-image: "ubuntu:22.04"
54-
latest-image: 'latest-gpu-hipblas-extras'
55-
latest-image-aio: 'latest-aio-gpu-hipblas'
56-
runs-on: 'arc-runner-set'
57-
makeflags: "--jobs=3 --output-sync=target"
5843
- build-type: 'hipblas'
5944
platforms: 'linux/amd64'
6045
tag-latest: 'false'
@@ -66,112 +51,8 @@ jobs:
6651
runs-on: 'arc-runner-set'
6752
makeflags: "--jobs=3 --output-sync=target"
6853
latest-image: 'latest-gpu-hipblas'
69-
self-hosted-jobs:
70-
uses: ./.github/workflows/image_build.yml
71-
with:
72-
tag-latest: ${{ matrix.tag-latest }}
73-
tag-suffix: ${{ matrix.tag-suffix }}
74-
ffmpeg: ${{ matrix.ffmpeg }}
75-
image-type: ${{ matrix.image-type }}
76-
build-type: ${{ matrix.build-type }}
77-
cuda-major-version: ${{ matrix.cuda-major-version }}
78-
cuda-minor-version: ${{ matrix.cuda-minor-version }}
79-
platforms: ${{ matrix.platforms }}
80-
runs-on: ${{ matrix.runs-on }}
81-
base-image: ${{ matrix.base-image }}
82-
grpc-base-image: ${{ matrix.grpc-base-image }}
83-
aio: ${{ matrix.aio }}
84-
makeflags: ${{ matrix.makeflags }}
85-
latest-image: ${{ matrix.latest-image }}
86-
latest-image-aio: ${{ matrix.latest-image-aio }}
87-
secrets:
88-
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
89-
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
90-
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
91-
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
92-
strategy:
93-
# Pushing with all jobs in parallel
94-
# eats the bandwidth of all the nodes
95-
max-parallel: ${{ github.event_name != 'pull_request' && 5 || 8 }}
96-
matrix:
97-
include:
98-
- build-type: 'cublas'
99-
cuda-major-version: "11"
100-
cuda-minor-version: "7"
101-
platforms: 'linux/amd64'
102-
tag-latest: 'false'
103-
tag-suffix: '-cublas-cuda11-extras'
104-
ffmpeg: 'true'
105-
image-type: 'extras'
106-
runs-on: 'arc-runner-set'
107-
base-image: "ubuntu:22.04"
108-
aio: "-aio-gpu-nvidia-cuda-11"
109-
latest-image: 'latest-gpu-nvidia-cuda-11-extras'
110-
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-11'
111-
makeflags: "--jobs=3 --output-sync=target"
112-
- build-type: 'cublas'
113-
cuda-major-version: "12"
114-
cuda-minor-version: "0"
115-
platforms: 'linux/amd64'
116-
tag-latest: 'false'
117-
tag-suffix: '-cublas-cuda12-extras'
118-
ffmpeg: 'true'
119-
image-type: 'extras'
120-
runs-on: 'arc-runner-set'
121-
base-image: "ubuntu:22.04"
122-
aio: "-aio-gpu-nvidia-cuda-12"
123-
latest-image: 'latest-gpu-nvidia-cuda-12-extras'
124-
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-12'
125-
makeflags: "--jobs=3 --output-sync=target"
126-
- build-type: 'sycl_f16'
127-
platforms: 'linux/amd64'
128-
tag-latest: 'false'
129-
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
130-
grpc-base-image: "ubuntu:22.04"
131-
tag-suffix: '-sycl-f16-extras'
132-
ffmpeg: 'true'
133-
image-type: 'extras'
134-
runs-on: 'arc-runner-set'
135-
aio: "-aio-gpu-intel-f16"
136-
latest-image: 'latest-gpu-intel-f16-extras'
137-
latest-image-aio: 'latest-aio-gpu-intel-f16'
138-
makeflags: "--jobs=3 --output-sync=target"
139-
- build-type: 'sycl_f32'
140-
platforms: 'linux/amd64'
141-
tag-latest: 'false'
142-
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
143-
grpc-base-image: "ubuntu:22.04"
144-
tag-suffix: '-sycl-f32-extras'
145-
ffmpeg: 'true'
146-
image-type: 'extras'
147-
runs-on: 'arc-runner-set'
148-
aio: "-aio-gpu-intel-f32"
149-
latest-image: 'latest-gpu-intel-f32-extras'
150-
latest-image-aio: 'latest-aio-gpu-intel-f32'
151-
makeflags: "--jobs=3 --output-sync=target"
152-
# Core images
153-
- build-type: 'sycl_f16'
154-
platforms: 'linux/amd64'
155-
tag-latest: 'false'
156-
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
157-
grpc-base-image: "ubuntu:22.04"
158-
tag-suffix: '-sycl-f16'
159-
ffmpeg: 'true'
160-
image-type: 'core'
161-
runs-on: 'arc-runner-set'
162-
makeflags: "--jobs=3 --output-sync=target"
163-
latest-image: 'latest-gpu-intel-f16'
164-
- build-type: 'sycl_f32'
165-
platforms: 'linux/amd64'
166-
tag-latest: 'false'
167-
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
168-
grpc-base-image: "ubuntu:22.04"
169-
tag-suffix: '-sycl-f32'
170-
ffmpeg: 'true'
171-
image-type: 'core'
172-
runs-on: 'arc-runner-set'
173-
makeflags: "--jobs=3 --output-sync=target"
174-
latest-image: 'latest-gpu-intel-f32'
54+
aio: "-aio-gpu-hipblas"
55+
latest-image-aio: 'latest-aio-gpu-hipblas'
17556

17657
core-image-build:
17758
uses: ./.github/workflows/image_build.yml
@@ -226,7 +107,9 @@ jobs:
226107
base-image: "ubuntu:22.04"
227108
makeflags: "--jobs=4 --output-sync=target"
228109
skip-drivers: 'false'
229-
latest-image: 'latest-gpu-nvidia-cuda-12'
110+
latest-image: 'latest-gpu-nvidia-cuda-11'
111+
aio: "-aio-gpu-nvidia-cuda-11"
112+
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-11'
230113
- build-type: 'cublas'
231114
cuda-major-version: "12"
232115
cuda-minor-version: "0"
@@ -240,6 +123,8 @@ jobs:
240123
skip-drivers: 'false'
241124
makeflags: "--jobs=4 --output-sync=target"
242125
latest-image: 'latest-gpu-nvidia-cuda-12'
126+
aio: "-aio-gpu-nvidia-cuda-12"
127+
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-12'
243128
- build-type: 'vulkan'
244129
platforms: 'linux/amd64'
245130
tag-latest: 'false'
@@ -251,6 +136,35 @@ jobs:
251136
skip-drivers: 'false'
252137
makeflags: "--jobs=4 --output-sync=target"
253138
latest-image: 'latest-gpu-vulkan'
139+
aio: "-aio-gpu-vulkan"
140+
latest-image-aio: 'latest-aio-gpu-vulkan'
141+
- build-type: 'sycl_f16'
142+
platforms: 'linux/amd64'
143+
tag-latest: 'false'
144+
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
145+
grpc-base-image: "ubuntu:22.04"
146+
tag-suffix: '-sycl-f16'
147+
ffmpeg: 'true'
148+
image-type: 'core'
149+
runs-on: 'arc-runner-set'
150+
makeflags: "--jobs=3 --output-sync=target"
151+
latest-image: 'latest-gpu-intel-f16'
152+
aio: "-aio-gpu-intel-f16"
153+
latest-image-aio: 'latest-aio-gpu-intel-f16'
154+
- build-type: 'sycl_f32'
155+
platforms: 'linux/amd64'
156+
tag-latest: 'false'
157+
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
158+
grpc-base-image: "ubuntu:22.04"
159+
tag-suffix: '-sycl-f32'
160+
ffmpeg: 'true'
161+
image-type: 'core'
162+
runs-on: 'arc-runner-set'
163+
makeflags: "--jobs=3 --output-sync=target"
164+
latest-image: 'latest-gpu-intel-f32'
165+
aio: "-aio-gpu-intel-f32"
166+
latest-image-aio: 'latest-aio-gpu-intel-f32'
167+
254168
gh-runner:
255169
uses: ./.github/workflows/image_build.yml
256170
with:

0 commit comments

Comments
 (0)