Skip to content

Commit c5e0d1b

Browse files
nicolereneefrezbo
authored andcommitted
feat: add service to create gdrdrv device
This is an extension to make it easier to use gdrcopy by creating the /dev/gdrdrv device for you. Signed-off-by: Nicole Hubbard <[email protected]> Signed-off-by: Noel Georgi <[email protected]>
1 parent b61799a commit c5e0d1b

File tree

12 files changed

+234
-1
lines changed

12 files changed

+234
-1
lines changed

.kres.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ spec:
4444
- nvidia-container-toolkit-production
4545
- nvidia-fabricmanager-lts
4646
- nvidia-fabricmanager-production
47+
- nvidia-gdrdrv-device
4748
- nvidia-open-gpu-kernel-modules-lts
4849
- nvidia-open-gpu-kernel-modules-production
4950
- nvme-cli

Makefile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# THIS FILE WAS AUTOMATICALLY GENERATED, PLEASE DO NOT EDIT.
22
#
3-
# Generated on 2025-11-07T09:07:26Z by kres 911d166.
3+
# Generated on 2025-11-11T07:27:41Z by kres 911d166.
44

55
# common variables
66

@@ -101,6 +101,7 @@ TARGETS += nvidia-container-toolkit-lts
101101
TARGETS += nvidia-container-toolkit-production
102102
TARGETS += nvidia-fabricmanager-lts
103103
TARGETS += nvidia-fabricmanager-production
104+
TARGETS += nvidia-gdrdrv-device
104105
TARGETS += nvidia-open-gpu-kernel-modules-lts
105106
TARGETS += nvidia-open-gpu-kernel-modules-production
106107
TARGETS += nvme-cli

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,7 @@ tiers based on support level:
175175
| [nvidia-container-toolkit-production](nvidia-gpu/nvidia-container-toolkit/production) | :green_square: core | [ghcr.io/siderolabs/nvidia-container-toolkit-production](https://github.com/siderolabs/extensions/pkgs/container/nvidia-container-toolkit-production) | `570.195.03-v1.18.0` | This system extension provides nvidia runtime and it's dependencies using NVIDIA's runtime handler. |
176176
| [nvidia-fabricmanager-lts](nvidia-gpu/nvidia-fabricmanager/lts) | :green_square: core | [ghcr.io/siderolabs/nvidia-fabricmanager-lts](https://github.com/siderolabs/extensions/pkgs/container/nvidia-fabricmanager-lts) | `580.95.05` | This system extension provides the Nvidia fabricmanager for GPU's that need NVLink support. |
177177
| [nvidia-fabricmanager-production](nvidia-gpu/nvidia-fabricmanager/production) | :green_square: core | [ghcr.io/siderolabs/nvidia-fabricmanager-production](https://github.com/siderolabs/extensions/pkgs/container/nvidia-fabricmanager-production) | `570.195.03` | This system extension provides the Nvidia fabricmanager for GPU's that need NVLink support. |
178+
| [nvidia-gdrdrv-device](nvidia-gpu/nvidia-gdrdrv-device) | :yellow_square: extra | [ghcr.io/siderolabs/nvidia-gdrdrv-device](https://github.com/siderolabs/extensions/pkgs/container/nvidia-gdrdrv-device) | `v2.5.1` | This system extension provides NVIDIA GPUDirect RDMA (gdrcopy) device file. The gdrdrv kernel module enables low-latency GPU memory access for RDMA operations. |
178179
| [nvidia-open-gpu-kernel-modules-lts](nvidia-gpu/nvidia-modules/lts) | :green_square: core | [ghcr.io/siderolabs/nvidia-open-gpu-kernel-modules-lts](https://github.com/siderolabs/extensions/pkgs/container/nvidia-open-gpu-kernel-modules-lts) | `580.95.05-VERSION` | This system extension provides nvidia open source driver kernel modules built against a specific Talos version. |
179180
| [nvidia-open-gpu-kernel-modules-production](nvidia-gpu/nvidia-modules/production) | :green_square: core | [ghcr.io/siderolabs/nvidia-open-gpu-kernel-modules-production](https://github.com/siderolabs/extensions/pkgs/container/nvidia-open-gpu-kernel-modules-production) | `570.195.03-VERSION` | This system extension provides nvidia open source driver kernel modules built against a specific Talos version. |
180181

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# NVIDIA GPUDirect RDMA (gdrcopy) Driver
2+
This extension provides the NVIDIA GPUDirect RDMA (gdrcopy) device for Talos Linux.
3+
## Description
4+
gdrcopy provide a low-latency GPU memory kernel-mode driver for NVIDIA GPUs. This enables:
5+
- Low-latency access to GPU memory from CPU
6+
- RDMA transfers to/from GPU memory
7+
- Support for GPUDirect RDMA technology
8+
The extension creates the `/dev/gdrdrv` device required to use gdrcopy. The kernel modules required are already bundled in the nvidia-open-gpu-kernel-modules.
9+
## Installation
10+
See [Installing Extensions](https://www.talos.dev/latest/talos-guides/configuration/system-extensions/) for general information on installing system extensions.
11+
## Usage
12+
After installing this extension, the `gdrdrv` kernel module will can be loaded by adding it to your machine config:
13+
```yaml
14+
machine:
15+
kernel:
16+
modules:
17+
- name: gdrdrv
18+
parameters:
19+
- dbg_enabled=0
20+
- info_enabled=0
21+
- use_persistent_mapping=1
22+
```
23+
The extension will then create the `/dev/gdrdrv` character device that applications can use to interact with GPU memory.
24+
## Requirements
25+
- NVIDIA GPU with GPUDirect RDMA support
26+
- nvidia-open-gpu-kernel-modules-production Talos extension
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
module gdrdrv-service
2+
3+
go 1.24.0
4+
5+
require golang.org/x/sys v0.37.0
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ=
2+
golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
// This Source Code Form is subject to the terms of the Mozilla Public
2+
// License, v. 2.0. If a copy of the MPL was not distributed with this
3+
// file, You can obtain one at http://mozilla.org/MPL/2.0/.
4+
package main
5+
6+
import (
7+
"bufio"
8+
"fmt"
9+
"log"
10+
"os"
11+
"strconv"
12+
"strings"
13+
"syscall" // Primarily for Mknod and Stat_t
14+
"time"
15+
16+
// Import the unix package from x/sys for Mkdev, Major, Minor
17+
// You'll need to run 'go get golang.org/x/sys/unix'
18+
"golang.org/x/sys/unix"
19+
)
20+
21+
const (
22+
deviceName = "gdrdrv" // The driver name we're looking for
23+
devicePath = "/dev/" + deviceName
24+
procDevicesPath = "/proc/devices" // File containing major numbers
25+
targetPerms = 0o666 // Desired file permissions (rw-rw-rw-)
26+
minorNumber = 0 // Standard minor number for gdrdrv
27+
retryDelay = 1 * time.Second // How long to wait between checks
28+
)
29+
30+
func findDeviceMajor() (int, error) {
31+
file, err := os.Open(procDevicesPath)
32+
if err != nil {
33+
return -1, fmt.Errorf("error opening %s: %w", procDevicesPath, err)
34+
}
35+
defer file.Close()
36+
37+
scanner := bufio.NewScanner(file)
38+
for scanner.Scan() {
39+
line := strings.TrimSpace(scanner.Text())
40+
fields := strings.Fields(line)
41+
42+
if len(fields) == 2 && fields[1] == deviceName {
43+
major, err := strconv.Atoi(fields[0])
44+
if err != nil {
45+
return -1, fmt.Errorf("error parsing major number '%s' for %s: %w", fields[0], deviceName, err)
46+
}
47+
return major, nil
48+
}
49+
}
50+
51+
if err := scanner.Err(); err != nil {
52+
return -1, fmt.Errorf("error reading %s: %w", procDevicesPath, err)
53+
}
54+
55+
return -1, fmt.Errorf("device '%s' not found in %s", deviceName, procDevicesPath)
56+
}
57+
58+
func main() {
59+
log.SetPrefix("gdrdrv-mknod: ")
60+
log.Printf("waiting for %s driver to register in %s...", deviceName, procDevicesPath)
61+
62+
var major int
63+
var err error
64+
65+
// Loop indefinitely until the major number is found
66+
for {
67+
major, err = findDeviceMajor()
68+
if err == nil {
69+
log.Printf("Found %s major number: %d", deviceName, major)
70+
break
71+
}
72+
// Log the error and wait before retrying
73+
log.Printf("waiting for device... (%v)", err)
74+
time.Sleep(retryDelay)
75+
}
76+
77+
// Attempt to remove the device node if it already exists
78+
log.Printf("checking for existing device node at %s...", devicePath)
79+
if _, err := os.Stat(devicePath); err == nil {
80+
log.Printf("removing existing %s...", devicePath)
81+
if err := os.Remove(devicePath); err != nil {
82+
log.Fatalf("failed to remove existing %s: %v", devicePath, err)
83+
}
84+
} else if !os.IsNotExist(err) {
85+
// If stat failed for a reason other than the file not existing, it's a problem
86+
log.Fatalf("error: checking status of %s: %v", devicePath, err)
87+
}
88+
89+
log.Printf("creating character device %s (Major: %d, Minor: %d)...", devicePath, major, minorNumber)
90+
91+
mode := uint32(syscall.S_IFCHR | targetPerms)
92+
93+
// Combine major and minor numbers into the format needed by Mknod
94+
// We use unix.Mkdev from the x/sys/unix package
95+
dev := unix.Mkdev(uint32(major), uint32(minorNumber))
96+
97+
// Execute the Mknod syscall
98+
if err := syscall.Mknod(devicePath, mode, int(dev)); err != nil {
99+
log.Fatalf("Failed to create device node with mknod: %v", err)
100+
}
101+
102+
// Mknod *should* set permissions, but explicitly call Chmod for robustness
103+
log.Printf("Setting permissions on %s to %#o...", devicePath, targetPerms)
104+
if err := os.Chmod(devicePath, targetPerms); err != nil {
105+
log.Printf("Warning: Failed to set permissions with chmod (mknod might have already set them): %v", err)
106+
}
107+
108+
// Verify creation and permissions by stating the file (like ls -l does)
109+
fileInfo, err := os.Stat(devicePath)
110+
if err != nil {
111+
log.Fatalf("Failed to stat the created device %s: %v", devicePath, err)
112+
}
113+
114+
// Extract major/minor from stat info for confirmation log
115+
devDetails := ""
116+
if stat_t, ok := fileInfo.Sys().(*syscall.Stat_t); ok {
117+
maj := unix.Major(uint64(stat_t.Rdev))
118+
min := unix.Minor(uint64(stat_t.Rdev))
119+
devDetails = fmt.Sprintf(" (Major: %d, Minor: %d)", maj, min)
120+
}
121+
122+
log.Printf("Successfully created %s: Mode=%s%s",
123+
devicePath,
124+
fileInfo.Mode().String(),
125+
devDetails,
126+
)
127+
}
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
version: v1alpha1
2+
metadata:
3+
name: nvidia-gdrdrv-device
4+
version: "{{ .VERSION }}"
5+
author: Nicole Hubbard
6+
description: |
7+
[{{ .TIER }}] This system extension provides NVIDIA GPUDirect RDMA (gdrcopy) device file.
8+
The gdrdrv kernel module enables low-latency GPU memory access for RDMA operations.
9+
compatibility:
10+
talos:
11+
version: ">= v1.10.0"
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: nvidia-gdrdrv-device
2+
depends:
3+
- service: cri
4+
container:
5+
entrypoint: /nvidia-gdrdrv-device
6+
mounts:
7+
- destination: /dev
8+
type: bind
9+
source: /dev
10+
options:
11+
- rshared
12+
- rbind
13+
- rw
14+
security:
15+
rootfsPropagation: shared
16+
restart: untilSuccess
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: nvidia-gdrdrv-device
2+
variant: scratch
3+
shell: /bin/bash
4+
dependencies:
5+
- stage: base
6+
steps:
7+
- env:
8+
GOPATH: /tmp/go
9+
- cachePaths:
10+
- /.cache/go-build
11+
- /tmp/go/pkg
12+
- network: default
13+
prepare:
14+
- |
15+
cp -r /pkg/* .
16+
- |
17+
go mod download
18+
- network: none
19+
build:
20+
- |
21+
CGO_ENABLED=0 go build -ldflags "-s -w" -trimpath -o nvidia-gdrdrv-device main.go
22+
install:
23+
- |
24+
mkdir -p /rootfs/usr/local/lib/containers/nvidia-gdrdrv-device
25+
cp nvidia-gdrdrv-device /rootfs/usr/local/lib/containers/nvidia-gdrdrv-device/
26+
mkdir -p /rootfs/usr/local/etc/containers
27+
cp /pkg/nvidia-gdrdrv-device.yaml /rootfs/usr/local/etc/containers/
28+
- test:
29+
- |
30+
mkdir -p /extensions-validator-rootfs
31+
cp -r /rootfs/ /extensions-validator-rootfs/rootfs
32+
cp /pkg/manifest.yaml /extensions-validator-rootfs/manifest.yaml
33+
/extensions-validator validate --rootfs=/extensions-validator-rootfs --pkg-name="${PKG_NAME}"
34+
finalize:
35+
- from: /rootfs
36+
to: /rootfs
37+
- from: /pkg/manifest.yaml
38+
to: /

0 commit comments

Comments
 (0)