Installing on different disk than bootloader

[jsawatzky]

I am attempting to get a Talos cluster running on a Turing Pi cluster using Turing RK1 compute modules. I have successfully booted the cluster but it did not install as I was expecting. I am looking for some clarifications on the config options.

Each of my 4 nodes has 32GB eMMC onboard and a 256GB NVMe drive attached. I flashed the Talos disk image onto the eMMC of each node and booted them. I then ran apply-config on each of the nodes with the following:

...
  install:
    disk: /dev/nvme0n1
    image: factory.talos.dev/metal-installer/85f683902139269fbc5a7f64ea94a694d31e0b3d94347a225223fcbd042083ae:v1.10.4
    wipe: true
...

I expected this to install the cluster onto the NVMe drives but running talosctl get discoveredvolumes revealed that this was not the case and everything was installed to the eMMC.

talosctl get disks output:

NODE             NAMESPACE   TYPE   ID             VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID                                                                                                       MODEL                    SERIAL
192.168.40.101   runtime     Disk   loop0          2         4.1 kB   true                                                                                                                                                                     
192.168.40.101   runtime     Disk   loop1          2         66 MB    true                                                                                                                                                                     
192.168.40.101   runtime     Disk   mmcblk0        2         31 GB    false       mmc                                                                                                                                                          0x5a87e879
192.168.40.101   runtime     Disk   mmcblk0boot0   2         4.2 MB   true        mmc                                                                                                                                                          
192.168.40.101   runtime     Disk   mmcblk0boot1   2         4.2 MB   true        mmc                                                                                                                                                          
192.168.40.101   runtime     Disk   nvme0n1        2         256 GB   false       nvme                     nvme.126f-503330304d424242323431313038303030313135-50617472696f74204d2e322050333030203235364742-00000001   Patriot M.2 P300 256GB   P300MBBB241108000115

talosctl get discoveredvolumes output:

NODE             NAMESPACE   TYPE               ID             VERSION   TYPE        SIZE     DISCOVERED   LABEL         PARTITIONLABEL
192.168.40.101   runtime     DiscoveredVolume   loop1          1         disk        66 MB    squashfs                   
192.168.40.101   runtime     DiscoveredVolume   mmcblk0        2         disk        31 GB    gpt                        
192.168.40.101   runtime     DiscoveredVolume   mmcblk0boot0   1         disk        4.2 MB                              
192.168.40.101   runtime     DiscoveredVolume   mmcblk0boot1   1         disk        4.2 MB                              
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p1      1         partition   105 MB   vfat         EFI           EFI
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p2      1         partition   1.0 MB                              BIOS
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p3      1         partition   1.0 GB   xfs          BOOT          BOOT
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p4      1         partition   1.0 MB                              META
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p5      2         partition   105 MB   xfs          STATE         STATE
192.168.40.101   runtime     DiscoveredVolume   mmcblk0p6      2         partition   30 GB    xfs          EPHEMERAL     EPHEMERAL
192.168.40.101   runtime     DiscoveredVolume   nvme0n1        1         disk        256 GB   gpt                        
192.168.40.101   runtime     DiscoveredVolume   nvme0n1p1      1         partition   537 MB   vfat         system-boot   primary
192.168.40.101   runtime     DiscoveredVolume   nvme0n1p2      1         partition   256 GB   ext4         writable      primary```

My question is how do I get Talos to install to the NVMe drives so I can leverage the larger storage? Do I need to do the initial install to the NVMe drives instead of flashing to eMMC?

Additionally, if it would be recommended to install Talos to the eMMC and utilize the NVMe storage in some other way, I am open to guidance on how best to configure my system.

Thank you in advance!

I have been following the following documentation:

- https://www.talos.dev/v1.10/talos-guides/install/single-board-computers/turing_rk1/
- https://www.talos.dev/v1.10/introduction/getting-started/#modifying-the-machine-configs

[smira]

I guess the confusion comes from the fact that Talos has never been installed actually, you just copied the disk image onto your mmcblk device, so Talos considers itself being installed.

So you have several options now:

• put the image on the NVMe drive instead of eMMC in the first place
• boot Talos e.g. over network (PXE) and then the install would trigger
• keep Talos boot on eMMC, including STATE partition, but move EPHEMERAL to NVMe: https://www.talos.dev/v1.10/talos-guides/configuration/disk-management/#ephemeral-volume

Please note that you would need to wipe existing installation before going with any of the options above.

[jsawatzky]

Thanks for the reply, that makes sense. It may be beneficial to add a note in the docs in regards to this case, as even the interactive mode of apply-config allowed me to select a disk that also wasn't used. For clarity for people like me in the future it would be useful to have a small notice indicating that the install disk will be ignored in this installation method.

I am curious if you have an opinion on which approach you would recommend between options 1 and 3 you have provided? I plan on deploying Longhorn for node storage if that makes a difference.

Thanks again for the help.

[smira]

I would recommend option (3), as some boards have trouble booting from NVME. I would allocate some NVMe for EPHEMERAL, e.g. 100 GiB (I guess that would be enough for emptyDir volumes and images?), and split the rest via user volumes for Longhorn, etc.

[jsawatzky]

Incredible, thank you so much for all the help and quick responses!

[dronkwors]

I was trying to do option 3, however by default EPHEMERAL is using /dev/mmcblk0p6 by default, I have tried to patch the cluster with the following:

apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
  diskSelector:
    match: disk.transport == 'nvme'
  maxSize: 100GB
  grow: false

I have even recreated the cluster from scratch with this config path applied to the machines before cluster creation. It feels like I am missing something...

[smira]

If you're using Omni, there's a known issue which makes EPHEMERAL contents being wiped, while the partition still stays. It will be addressed in Talos 1.12, or you can simply wipe the SD card, put a new disk image on it, and try with the path with the initial installation.