Waiting for service ext-zfs-service to be up

[alongwill]

I have a VM in Civo Cloud that was:

1. Created as an Ubuntu 24.04 VM in Civo Cloud

VM disks and network details

Disks

root@talos-cp-0:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda     253:0    0 47.3G  0 disk
├─vda1  253:1    0 46.2G  0 part /
├─vda14 253:14   0    4M  0 part
├─vda15 253:15   0  106M  0 part /boot/efi
└─vda16 259:0    0  913M  0 part /boot
vdb     253:16   0    1M  0 disk

Network

root@talos-cp-0:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 42:7f:94:c1:24:d5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 metric 100 brd 192.168.1.255 scope global dynamic enp1s0
       valid_lft 86313480sec preferred_lft 86313480sec
    inet6 fe80::407f:94ff:fec1:24d5/64 scope link
       valid_lft forever preferred_lft forever

2. Re-installed with Talos 1.10.6 using boot-to-talos

boot-to-talos command logs

root@talos-cp-0:~# boot-to-talos -image ghcr.io/cozystack/cozystack/talos:v1.10.6 \
  -extra-kernel-arg 'siderolink.api=https://****.siderolink.eu-central-1.omni.siderolabs.io?jointoken=****' \
  -extra-kernel-arg 'talos.events.sink=[fdae:41e4:649b:9303::1]:8090' \
  -extra-kernel-arg 'talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092'
Target disk [/dev/vda]:
Add networking configuration? [yes]:
Interface [enp1s0]:
IP address [192.168.1.2]:
Netmask [255.255.255.0]:
Gateway (or 'none') [192.168.1.1]:
Configure serial console? (or 'no') [ttyS0]:

Summary:
  Image: ghcr.io/cozystack/cozystack/talos:v1.10.6
  Disk:  /dev/vda
  Extra kernel args: siderolink.api=https://****.siderolink.eu-central-1.omni.siderolabs.io?jointoken=**** talos.events.sink=[fdae:41e4:649b:9303::1]:8090 talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092 ip=192.168.1.2::192.168.1.1:255.255.255.0::enp1s0::::: console=ttyS0

WARNING: ALL DATA ON /dev/vda WILL BE ERASED!

Continue? [yes]: yes

2025/10/01 09:46:58 created temporary directory /tmp/installer-3484253475
2025/10/01 09:46:58 pulling image ghcr.io/cozystack/cozystack/talos:v1.10.6
2025/10/01 09:46:58 No proxy configured, using direct connection
2025/10/01 09:46:59 extracting image layers
2025/10/01 09:46:59 No proxy configured, using direct connection
2025/10/01 09:47:08 creating raw disk /tmp/installer-3484253475/image.raw (2 GiB)
2025/10/01 09:47:08 attached /tmp/installer-3484253475/image.raw to /dev/loop0
2025/10/01 09:47:08 starting Talos installer
2025/10/01 09:47:08 running Talos installer v1.10.6
2025/10/01 09:47:08 WARNING: config validation:
2025/10/01 09:47:08   use "worker" instead of "" for machine type
2025/10/01 09:47:08 created EFI (C12A7328-F81F-11D2-BA4B-00A0C93EC93B) size 104857600 bytes
2025/10/01 09:47:08 created BIOS (21686148-6449-6E6F-744E-656564454649) size 1048576 bytes
2025/10/01 09:47:08 created BOOT (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576000 bytes
2025/10/01 09:47:08 created META (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576 bytes
2025/10/01 09:47:08 formatting the partition "/dev/loop0p1" as "vfat" with label "EFI"
2025/10/01 09:47:08 formatting the partition "/dev/loop0p2" as "zeroes" with label "BIOS"
2025/10/01 09:47:08 formatting the partition "/dev/loop0p3" as "xfs" with label "BOOT"
2025/10/01 09:47:08 formatting the partition "/dev/loop0p4" as "zeroes" with label "META"
2025/10/01 09:47:08 copying from io reader to /boot/A/vmlinuz
2025/10/01 09:47:08 copying from io reader to /boot/A/initramfs.xz
2025/10/01 09:47:08 writing /boot/grub/grub.cfg to disk
2025/10/01 09:47:08 executing: grub-install --boot-directory=/boot --removable --efi-directory=/boot/EFI /dev/loop0
2025/10/01 09:47:09 installation of v1.10.6 complete
2025/10/01 09:47:09 Talos installer finished successfully
2025/10/01 09:47:09 remounting all filesystems read-only
2025/10/01 09:47:09 copy /tmp/installer-3484253475/image.raw → /dev/vda
2025/10/01 09:47:39 installation image copied to /dev/vda
2025/10/01 09:47:39 rebooting system

This machine connects to Omni successfully. I add it to a cluster and everything starts fine except the zfs service. The machine shows as Booting in Omni and the console logs show repeatedly:

[talos] task startAllServices (1/1): service "ext-zfs-service" to be "up"

talosctl services shows:

talosctl services -n talos-533-82w
NODE          SERVICE                STATE     HEALTH   LAST CHANGE   LAST EVENT
192.168.1.2   apid                   Running   OK       21m36s ago    Health check successful
192.168.1.2   auditd                 Running   OK       21m38s ago    Health check successful
192.168.1.2   containerd             Running   OK       21m38s ago    Health check successful
192.168.1.2   cri                    Running   OK       21m35s ago    Health check successful
192.168.1.2   dashboard              Running   ?        21m37s ago    Process Process(["/sbin/dashboard"]) started with PID 2137
192.168.1.2   etcd                   Running   OK       21m29s ago    Health check successful
192.168.1.2   ext-zfs-service        Waiting   ?        21m34s ago    Waiting for file "/dev/zfs" to exist
192.168.1.2   kubelet                Running   OK       21m27s ago    Health check successful
192.168.1.2   machined               Running   OK       21m38s ago    Health check successful
192.168.1.2   syslogd                Running   OK       21m37s ago    Health check successful
192.168.1.2   trustd                 Running   OK       21m35s ago    Health check successful
192.168.1.2   udevd                  Running   OK       21m38s ago    Health check successful

I tried using an installer with the zfs extension but that resulted in the same issue.

How can I clear the service "ext-zfs-service" to be "up" issue?

[alongwill]

Ah. Added this machine config and that's cleared the issue.

      machine:
        kernel:
          modules:
            - name: zfs