Bare Metal Install does not survive reboot

[imnotjames]

I am currently trying to get a bare metal install working on a Dell Poweredge 730XD.

I am using the factory metal-amd64.iso booted off a USB drive, with nothing special added as far as I'm aware.

It boots up fine from the ISO but after running apply-config I see the following error when trying to reboot and use the disk:

The machine portion of the YAML is as follows.

machine:
    type: controlplane
    token: REDACTED
    ca:
        crt: REDACTED
        key: REDACTED
    certSANs: []
    kubelet:
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    network: {}
    install:
        disk: /dev/sdb
        wipe: true
    registries: {}
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
        diskQuotaSupport: true
        kubePrism:
            enabled: true
            port: 7445
        hostDNS:
            enabled: true
            forwardKubeDNSToHost: true

It's unclear what is going on here - the failure seems to be in mounting the rootfs via /dev/loop0. I've tried wiping the disk entirely with another live CD, I've tries the wipe: true, I just can't seem to get things to boot properly.

I have no issues with installing debian on the same disk, and SMART shows it as healthy.

Is there any extra debugging information I could retrieve to point me in the right direction?

---

To clarify what I have done:

• I download the ISO from the image factory at factory.talos.dev
• I write that image to a USB drive via dd
• Boot from the drive and wait for the STAGE to be MAINTENANCE and READY to be True
• I run the command kubectl apply-config --insecure --nodes $CONTROL_PLANE_IP --file controlplane.yaml
• This causes a reboot of the system once it completes and writes to /dev/sda
• I use the boot manager from the BIOS to boot from the local drive and skip the USB
• At that point I see the error message and an automatic reboot occurs

I've also tried pulling the USB drive after the initial ISO boot - same issue.

[frezbo]

From the console logs it seems secureboot is enabled, but then metal-amd64.iso is not the secureboot iso, so I'm not sure what got booted here, did you boot the Talos secureboot image?

[imnotjames]

I might have mixed up the filename:

• I went to factory.talos.dev
• Selected Bare Metal
• Selected 1.10.6
• Selected AMD64 + SecureBoot architecture
• Omitted all extensions
• Skipped any extra parameters
• Download the ISO and used dd to write it to the USB installation drive.

I have, however, tried the non-secureboot version directly from the Releases tab on github - in case it was something off with UEFI and secureboot, but had the same message.

[rothgar]

If you went through an install once and then boot a new iso there's a menu to wipe the existing install. If you don't do that I don't think the system will go back into maintenance mode. If you try the 1.11 RC release you can also wipe the disk from maintenance mode now.

Things I would check:

• secure boot is enabled in your system UEFI
• key enrollment is enabled
• Maybe also try disabling secure boot and booting from a non-secure boot iso

[imnotjames]

Thanks for the response.

If you went through an install once and then boot a new iso there's a menu to wipe the existing install. If you don't do that I don't think the system will go back into maintenance mode. If you try the 1.11 RC release you can also wipe the disk from maintenance mode now.

Yeah - I've gotten super familiar with that :) I've been trying lots of different versions and wipes. I did use the "wipe system disk" feature, but also have wiped using ShredOS and other mechanisms.

Things I would check:

* secure boot is enabled in your system UEFI

Yes, I have been able to confirm that Secure boot is enabled.

* key enrollment is enabled

Yes, key enrollment is enabled.

* Maybe also try disabling secure boot and booting from a non-secure boot iso

I'll definitely double check this and try it.

The system DID boot. The UEFI bootloader started up the initial linux kernel, but when trying to mount the squashfs filesystem (to root, I think?) is where things failed to run as expected.

[imnotjames]

I've tried 1.11 RC & 1.10.5, still the same problem.

I tried 1.9.6, that installs and runs fine - no problems. So maybe something to do with a change between 1.9.6 & 1.10.0?

[imnotjames]

Per @rothgar I disabled UEFI and installed Talos v1.10.6 - and that succeeds in fully booting up after a reboot. I am assuming that because UEFI booting is not available that GRUB is used instead which allows for the squashfs image to be mounted.

[rothgar]

We will still maintain the GRUB boot path on non-UEFI systems. The goal is to automatically use systemd-boot for UEFI systems.

@frezbo may have more information to share and if we could get more detailed logs we might be able to figure out why this is happening on this hardware

[frezbo]

this feels more like a corruption, nothing with how talos is booted, unless the UEFI firmware has some weird bug

[rothgar]

Maybe try updating the system firmware?

[imnotjames]

if we could get more detailed logs we might be able to figure out why this is happening on this hardware

Unfortunately, because this happens before everything is fully booted I can't run any commands or get text logs. If there are ways to get more logs I'm happy to do it and capture them via screenshot? Or if there's a way to tell systemd-boot to write logs to the disk I will. Not sure off hand how to do that, though.

I also struggled to get logs during the install as the system restarts as soon as the install completes. Perhaps I can tail the logs during the install somehow into a file on a remote computer? I'll give that a shot..

this feels more like a corruption, nothing with how talos is booted, unless the UEFI firmware has some weird bug

That's possible. I've tried 3 different USB drives and 2 SSDs.

The same image that is working now with legacy BIOS enabled was the same one I used with UEFI and saw the issue.

Maybe try updating the system firmware?

The Dell Poweredge 730XD is running 2.19.0 which seems to be the most recent firmware available. I also had seen the same issue on 2.17.0

[rothgar]

You should be able to get text console logs from the system via IPMI. That should show the entire boot process and crashes.