Skip to content

Commit 55b18d0

Browse files
rollandfrollandf
authored
Merge pull request #58 from e0ne/precompiled-driver
Added precompiled container build instructions for DOCA drivers
2 parents d1e245d + bc54862 commit 55b18d0

File tree

5 files changed

+1524
-0
lines changed

5 files changed

+1524
-0
lines changed

‎docs/advanced-configurations.rst‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -304,3 +304,123 @@ If self-signed certificates are used for an HTTPS based internal repository, a C
304304
deploy: true
305305
certConfg:
306306
name: cert-config
307+
308+
=========================================================
309+
Precompiled Container Build Instructions for DOCA Drivers
310+
=========================================================
311+
312+
-------------
313+
Prerequisites
314+
-------------
315+
316+
Before you begin, ensure that you have the following prerequisites:
317+
318+
~~~~~~
319+
Common
320+
~~~~~~
321+
322+
- Docker (Ubuntu) / Podman (RH) installed on your build system.
323+
- Web access to NVIDIA NIC drivers sources. Latest NIC drivers published at `NIC drivers download center <https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/>`_, for example: `https://www.mellanox.com/downloads/ofed/MLNX_OFED-24.04-0.6.6.0/MLNX_OFED_SRC-debian-24.04-0.6.6.0-0.tgz <https://www.mellanox.com/downloads/ofed/MLNX_OFED-24.04-0.6.6.0/MLNX_OFED_SRC-debian-24.04-0.6.6.0-0.tgz>`_
324+
325+
326+
~~~~
327+
RHEL
328+
~~~~
329+
330+
- Active subscription and login credentials for `registry.redhat.io <https://registry.redhat.io>`_. To build RHEL based container from official repository, you need to log in to `registry.redhat.io <https://registry.redhat.io>`_, run the following command:
331+
332+
.. code-block:: bash
333+
334+
podman login registry.redhat.io --username=${RH_USERNAME} --password=${RH_PASSWORD}
335+
336+
Replace `RH_USERNAME` and `RH_PASSWORD` with your Red Hat account username and password.
337+
338+
-------------------
339+
Dockerfile Overview
340+
-------------------
341+
342+
To build the precompiled container, the Dockerfile is constructed in a multistage fashion.
343+
This approach is used to optimize the resulting container image size and reduce the number of dependencies included in the final image.
344+
345+
The Dockerfile consists of the following stages:
346+
347+
1. **Base Image Update**: The base image is updated and common requirements are installed. This stage sets up the basic environment for the subsequent stages.
348+
349+
2. **Download Driver Sources**: This stage downloads the Mellanox OFED driver sources to the specified path. It prepares the necessary files for the driver build process.
350+
351+
3. **Build Driver**: The driver is built using the downloaded sources and installed on the container. This stage ensures that the driver is compiled and configured correctly for the target system.
352+
353+
4. **Install precompiled driver**: Finally, the precompiled driver is installed on clean container. This stage sets up the environment to run the NVIDIA NIC drivers on the target system.
354+
355+
356+
---------------------------------
357+
Common mandatory build parameters
358+
---------------------------------
359+
360+
Before building the container, you need to provide following parameters as `build-arg` for container build:
361+
362+
1. `D_OS`: The Linux distribution (e.g., ubuntu22.04 / rhel9.2)
363+
2. `D_ARCH`: Compiled Architecture
364+
3. `D_BASE_IMAGE`: Base container image
365+
4. `D_KERNEL_VER`: The target kernel version (e.g., 5.15.0-25-generic / 5.14.0-284.32.1.el9_2.x86_64)
366+
5. `D_OFED_VERSION`: NVIDIA NIC drivers version (e.g., 24.01-0.3.3.1)
367+
368+
**NOTE:** Check desired NVIDIA NIC drivers sources[^1] availability for designated container OS, only versions available on download page can be utilized
369+
370+
------------------------------
371+
RHEL-specific build parameters
372+
------------------------------
373+
374+
1. `D_BASE_IMAGE`: DriverToolKit container image
375+
376+
**NOTE:** DTK (DriverToolKit) is tightly coupled with specific kernel versions, verify match between kernel version to compile drivers for, versus DTK image.
377+
378+
2. `D_FINAL_BASE_IMAGE`: Final container image, to install compiled driver
379+
380+
For more details regarding DTK please read `official documentation <https://docs.openshift.com/container-platform/4.15/hardware_enablement/psap-driver-toolkit.html#pulling-the-driver-toolkit-from-payload>`_.
381+
382+
**NOTE:** For proper Network Operator functionality container tag name must be in following pattern: **driver_ver-container_ver-kernel_ver-os-arch**. For example: 24.01-0.3.3.1-0-5.15.0-25-generic-ubuntu22.04-amd64
383+
384+
~~~~~~~~~~~~
385+
RHEL example
386+
~~~~~~~~~~~~
387+
388+
To build RHEL-based image please use provided :download:`Dockerfile <files/RHEL_Dockerfile>`:
389+
390+
.. code-block:: bash
391+
392+
podman build \
393+
--build-arg D_OS=rhel9.2 \
394+
--build-arg D_ARCH=x86_64 \
395+
--build-arg D_KERNEL_VER=5.14.0-284.32.1.el9_2.x86_64 \
396+
--build-arg D_OFED_VERSION=24.01-0.3.3.1 \
397+
--build-arg D_BASE_IMAGE="registry.redhat.io/openshift4/driver-toolkit-rhel9:v4.13.0-202309112001.p0.gd719bdc.assembly.stream" \
398+
--build-arg D_FINAL_BASE_IMAGE=registry.access.redhat.com/ubi9/ubi:latest \
399+
--tag 24.04-0.6.6.0-0-5.14.0-284.32.1.el9_2-rhel9.2-amd64 \
400+
-f RHEL_Dockerfile \
401+
--target precompiled .
402+
403+
~~~~~~~~~~~~~~
404+
Ubuntu example
405+
~~~~~~~~~~~~~~
406+
407+
To build RHEL-based image please use provided :download:`Dockerfile <files/Ubuntu_Dockerfile>`:.
408+
409+
.. code-block:: bash
410+
411+
docker build \
412+
--build-arg D_OS=ubuntu22.04 \
413+
--build-arg D_ARCH=x86_64 \
414+
--build-arg D_BASE_IMAGE=ubuntu:24.04 \
415+
--build-arg D_KERNEL_VER=5.15.0-25-generic \
416+
--build-arg D_OFED_VERSION=24.01-0.3.3.1 \
417+
--tag 24.01-0.3.3.1-0-5.15.0-25-generic-ubuntu22.04-amd64 \
418+
-f Ubuntu_Dockerfile \
419+
--target precompiled .
420+
421+
**NOTE:** Dockerfiles contain default build parameters, which may fail build proccess on your system if not overridden.
422+
423+
**NOTE:** Entrypoint script :download:`download <files/entrypoint.sh>`
424+
**NOTE:** Driver build script :download:`download <files/dtk_nic_driver_build.sh>`
425+
426+
.. warning:: Modification of `D_OFED_SRC_DOWNLOAD_PATH` must be tighdly coupled with corresponding update to entrypoint.sh script.

‎docs/files/RHEL_Dockerfile‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Common (multistage) args
2+
ARG D_OS="rhel9.2"
3+
ARG D_ARCH="x86_64"
4+
ARG D_CONTAINER_VER="0"
5+
ARG D_OFED_VERSION="24.04-0.6.6.0"
6+
ARG D_KERNEL_VER="5.14.0-284.32.1.el9_2.x86_64"
7+
ARG D_OFED_SRC_DOWNLOAD_PATH="/run/mellanox/src"
8+
ARG OFED_SRC_LOCAL_DIR=${D_OFED_SRC_DOWNLOAD_PATH}/MLNX_OFED_SRC-${D_OFED_VERSION}
9+
10+
# Final clean image of precompiled driver container
11+
ARG D_FINAL_BASE_IMAGE=registry.access.redhat.com/ubi9/ubi:latest
12+
13+
##################################################################
14+
# Stage: Minimal base image update and install common requirements
15+
16+
# DTK base image (below example for specific kernel headers version)
17+
ARG D_BASE_IMAGE="registry.redhat.io/openshift4/driver-toolkit-rhel9:v4.13.0-202309112001.p0.gd719bdc.assembly.stream"
18+
# Standart: registry.access.redhat.com/ubi9:latest
19+
20+
ARG D_PYTHON_VERSION="36"
21+
ARG D_PYTHON="python${D_PYTHON_VERSION}"
22+
23+
FROM $D_BASE_IMAGE AS base
24+
25+
# Inherited global args
26+
ARG D_OS
27+
28+
RUN if [[ "${D_OS}" == *"rhel9"* ]] ; then \
29+
sed -i 's#/etc/pki/entitlement#/etc/pki/entitlement-host#g' /etc/rhsm/rhsm.conf ;\
30+
fi
31+
32+
RUN set -x && \
33+
# Driver build / install script requirements
34+
dnf -y install perl \
35+
# Container functional requirements
36+
jq iproute kmod procps-ng udev
37+
38+
##############################################################################################
39+
# Stage: Download NVIDIA driver sources and install src driver container packages requirements
40+
41+
FROM base AS driver-src
42+
43+
# Inherited global args
44+
ARG D_OFED_VERSION
45+
ARG D_CONTAINER_VER
46+
ARG D_OFED_SRC_DOWNLOAD_PATH
47+
48+
# Stage args
49+
ARG D_OFED_BASE_URL="https://www.mellanox.com/downloads/ofed/MLNX_OFED-${D_OFED_VERSION}"
50+
ARG D_OFED_SRC_TYPE=""
51+
52+
ARG D_OFED_SRC_ARCHIVE="MLNX_OFED_SRC-${D_OFED_SRC_TYPE}${D_OFED_VERSION}.tgz"
53+
ARG D_OFED_URL_PATH="${D_OFED_BASE_URL}/${D_OFED_SRC_ARCHIVE}"
54+
55+
ENV NVIDIA_NIC_DRIVER_VER=${D_OFED_VERSION}
56+
ENV NVIDIA_NIC_CONTAINER_VER=${D_CONTAINER_VER}
57+
ENV NVIDIA_NIC_DRIVER_PATH="${D_OFED_SRC_DOWNLOAD_PATH}/MLNX_OFED_SRC-${D_OFED_VERSION}"
58+
59+
WORKDIR /root
60+
RUN set -x && \
61+
# Install prerequirements
62+
dnf install -y curl --allowerasing \
63+
# Driver build requirements
64+
autoconf python3-devel ethtool automake pciutils libtool hostname
65+
66+
RUN set -x && \
67+
# Download NVIDIA NIC driver sources
68+
mkdir -p ${D_OFED_SRC_DOWNLOAD_PATH} && \
69+
cd ${D_OFED_SRC_DOWNLOAD_PATH} && (curl -sL ${D_OFED_URL_PATH} | tar -xzf -)
70+
71+
WORKDIR /
72+
ADD ./entrypoint.sh /root/entrypoint.sh
73+
ADD ./dtk_nic_driver_build.sh /root/dtk_nic_driver_build.sh
74+
75+
ENTRYPOINT ["/root/entrypoint.sh"]
76+
CMD ["sources"]
77+
78+
#####################
79+
# Stage: Build driver
80+
81+
FROM driver-src AS driver-builder
82+
83+
# Inherited global args
84+
ARG D_OS
85+
ARG D_KERNEL_VER
86+
ARG OFED_SRC_LOCAL_DIR
87+
88+
RUN set -x && \
89+
# MOFED installation requirements
90+
dnf install -y autoconf gcc make rpm-build
91+
92+
# Build driver
93+
RUN set -x && \
94+
${OFED_SRC_LOCAL_DIR}/install.pl --without-depcheck --distro ${D_OS} --kernel ${D_KERNEL_VER} --kernel-sources /lib/modules/${D_KERNEL_VER}/build --kernel-only --build-only --without-iser --without-srp --without-isert --without-knem --without-xpmem --with-mlnx-tools --with-ofed-scripts --copy-ifnames-udev
95+
96+
###################################
97+
# Stage: Install precompiled driver
98+
99+
ARG D_FINAL_BASE_IMAGE
100+
101+
FROM $D_FINAL_BASE_IMAGE AS precompiled
102+
103+
# Inherited global args
104+
ARG D_ARCH
105+
ARG D_KERNEL_VER
106+
ARG D_OFED_VERSION
107+
ARG D_CONTAINER_VER
108+
ARG OFED_SRC_LOCAL_DIR
109+
110+
ENV NVIDIA_NIC_DRIVER_VER=${D_OFED_VERSION}
111+
ENV NVIDIA_NIC_DRIVER_PATH=""
112+
ENV NVIDIA_NIC_CONTAINER_VER=${D_CONTAINER_VER}
113+
114+
COPY --from=driver-builder ${OFED_SRC_LOCAL_DIR}/RPMS/redhat-release-*/${D_ARCH}/*.rpm /root/
115+
116+
WORKDIR /root/
117+
RUN set -x && \
118+
rpm -ivh --nodeps \
119+
./kmod-mlnx-nfsrdma-*.rpm \
120+
./kmod-mlnx-nvme-*.rpm \
121+
./kmod-mlnx-ofa_kernel-*.rpm \
122+
./mlnx-ofa_kernel-*.rpm \
123+
./mlnx-tools-*.rpm
124+
125+
RUN set -x && \
126+
# MOFED functional requirements
127+
dnf install -y pciutils hostname udev ethtool \
128+
# Container functional requirements
129+
jq iproute kmod procps-ng udev
130+
131+
# Prevent modprobe from giving a WARNING about missing files
132+
RUN touch /lib/modules/${D_KERNEL_VER}/modules.order /lib/modules/${D_KERNEL_VER}/modules.builtin && \
133+
# Introduce installed kernel modules
134+
depmod ${D_KERNEL_VER}
135+
136+
WORKDIR /
137+
ADD ./entrypoint.sh /root/entrypoint.sh
138+
ADD ./dtk_nic_driver_build.sh /root/dtk_nic_driver_build.sh
139+
140+
ENTRYPOINT ["/root/entrypoint.sh"]
141+
CMD ["precompiled"]

‎docs/files/Ubuntu_Dockerfile‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Common (multistage) args
2+
ARG D_OS="ubuntu22.04"
3+
ARG D_ARCH="x86_64"
4+
ARG D_CONTAINER_VER="0"
5+
ARG D_OFED_VERSION="24.04-0.6.6.0"
6+
ARG D_KERNEL_VER="5.15.0-25-generic"
7+
ARG D_OFED_SRC_DOWNLOAD_PATH="/run/mellanox/src"
8+
ARG OFED_SRC_LOCAL_DIR=${D_OFED_SRC_DOWNLOAD_PATH}/MLNX_OFED_SRC-${D_OFED_VERSION}
9+
10+
# Common for build and final clean image of precompiled driver container
11+
ARG D_BASE_IMAGE="ubuntu:22.04"
12+
13+
##################################################################
14+
# Stage: Minimal base image update and install common requirements
15+
FROM $D_BASE_IMAGE AS base
16+
17+
ARG D_APT_REMOVE=""
18+
ARG D_OFED_VERSION
19+
ARG D_CONTAINER_VER
20+
ARG D_OFED_SRC_DOWNLOAD_PATH
21+
22+
ENV NVIDIA_NIC_DRIVER_VER=${D_OFED_VERSION}
23+
ENV NVIDIA_NIC_CONTAINER_VER=${D_CONTAINER_VER}
24+
25+
WORKDIR /root
26+
RUN set -x && \
27+
for source in ${D_APT_REMOVE}; do rm -f /etc/apt/sources.list.d/${source}.list; done && \
28+
# Perform distro update and install prerequirements
29+
apt-get -yq update && \
30+
DEBIAN_FRONTEND=noninteractive apt-get -yq upgrade && \
31+
DEBIAN_FRONTEND=noninteractive apt-get -yq install apt-utils \
32+
# Driver build / install script requirements
33+
perl pciutils kmod lsof python3 dh-python \
34+
# Container functional requirements
35+
jq iproute2 udev ethtool
36+
37+
WORKDIR /
38+
ADD ./entrypoint.sh /root/entrypoint.sh
39+
40+
ENTRYPOINT ["/root/entrypoint.sh"]
41+
42+
##############################################################################################
43+
# Stage: Download NVIDIA driver sources and install src driver container packages requirements
44+
45+
FROM base AS driver-src
46+
47+
# Inherited global args
48+
ARG D_OFED_VERSION
49+
ARG D_OFED_SRC_DOWNLOAD_PATH
50+
51+
# Stage args
52+
ARG D_OFED_BASE_URL="https://www.mellanox.com/downloads/ofed/MLNX_OFED-${D_OFED_VERSION}"
53+
ARG D_OFED_SRC_TYPE="debian-"
54+
55+
ARG D_OFED_SRC_ARCHIVE="MLNX_OFED_SRC-${D_OFED_SRC_TYPE}${D_OFED_VERSION}.tgz"
56+
ARG D_OFED_URL_PATH="${D_OFED_BASE_URL}/${D_OFED_SRC_ARCHIVE}"
57+
58+
ENV NVIDIA_NIC_DRIVER_PATH="${D_OFED_SRC_DOWNLOAD_PATH}/MLNX_OFED_SRC-${D_OFED_VERSION}"
59+
60+
WORKDIR /root
61+
RUN set -x && \
62+
# Install prerequirements
63+
DEBIAN_FRONTEND=noninteractive apt-get -yq install curl \
64+
dkms make autoconf autotools-dev chrpath automake hostname debhelper gcc quilt libc6-dev build-essential pkg-config && \
65+
# Cleanup
66+
apt-get clean autoclean && \
67+
rm -rf /var/lib/apt/lists/*
68+
69+
RUN set -x && \
70+
# Download NVIDIA NIC driver sources
71+
mkdir -p ${D_OFED_SRC_DOWNLOAD_PATH} && \
72+
cd ${D_OFED_SRC_DOWNLOAD_PATH} && (curl -sL ${D_OFED_URL_PATH} | tar -xzf -)
73+
74+
CMD ["sources"]
75+
76+
#####################
77+
# Stage: Build driver
78+
79+
FROM driver-src AS driver-builder
80+
81+
# Inherited global args
82+
ARG D_OS
83+
ARG D_KERNEL_VER
84+
ARG OFED_SRC_LOCAL_DIR
85+
86+
# Driver build manadatory packages
87+
RUN set -x && \
88+
apt-get update && \
89+
DEBIAN_FRONTEND=noninteractive apt-get -yq install linux-image-${D_KERNEL_VER} linux-headers-${D_KERNEL_VER}
90+
91+
# Build driver
92+
RUN set -x && \
93+
${OFED_SRC_LOCAL_DIR}/install.pl --without-depcheck --distro ${D_OS} --without-dkms --kernel ${D_KERNEL_VER} --kernel-only --build-only --copy-ifnames-udev --with-mlnx-tools --without-knem-modules --without-srp-modules --without-kernel-mft-modules --without-iser-modules --without-isert-modules
94+
95+
###################################
96+
# Stage: Install precompiled driver
97+
98+
FROM base AS precompiled
99+
100+
# Inherited global args
101+
ARG D_OS
102+
ARG D_ARCH
103+
ARG D_KERNEL_VER
104+
ARG OFED_SRC_LOCAL_DIR
105+
106+
ENV NVIDIA_NIC_DRIVER_PATH=""
107+
108+
RUN set -x && \
109+
apt-get install -y lsb-release && \
110+
# Cleanup
111+
apt-get clean autoclean && \
112+
rm -rf /var/lib/apt/lists/*
113+
114+
# Install driver
115+
COPY --from=driver-builder ${OFED_SRC_LOCAL_DIR}/DEBS/${D_OS}/${D_ARCH}/*.deb /root/
116+
RUN dpkg -i /root/*.deb
117+
118+
# Prevent modprobe from giving a WARNING about missing files
119+
RUN touch /lib/modules/${D_KERNEL_VER}/modules.order /lib/modules/${D_KERNEL_VER}/modules.builtin && \
120+
# Introduce installed kernel modules
121+
depmod ${D_KERNEL_VER}
122+
123+
CMD ["precompiled"]

0 commit comments

Comments
 (0)