rook-ceph Reset OSD

[ktoulliou]

I can't wipe my SAS disks (non-RAID devices of type megaraid_sas) after delete and reinstall rook-ceph. I don't know if it's Talos problem or just rook-ceph ?

But NVME is compatible with this sanitization procedure.

Procedure:

• patch ceph-cluster + remove ceph-cluster + delete operator + remove namespace
• on each disk : sgdisk --zapp-all + dd + blkdiscard + partprobe
• remove /var/lib/rook on each node
• reinstall rook operator and ceph cluster

Some metadata resists this sanitization procedure for SAS disks :

2025-09-19 13:07:26.943486 I | cephosd: skipping osd.9: "dc33bdae-16a7-49ba-8cc4-236b6d4fdcc5" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.943504 I | cephosd: skipping osd.106: "5a58d298-8f39-4dd7-bd41-6424d7d274db" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.943516 I | cephosd: skipping osd.81: "5b0da6b9-2489-4aef-9519-b140574ad2e3" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.943527 I | cephosd: skipping osd.34: "7c9578db-6e61-40cf-807d-5927e5c41be7" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958341 I | cephosd: setting device type "nvme" for device "/dev/nvme2n1"
2025-09-19 13:07:26.958356 I | cephosd: setting device class "nvme" for device "/dev/nvme2n1"
2025-09-19 13:07:26.958375 I | cephosd: skipping osd.43: "d8344033-2583-4e91-b9ef-03b59877c213" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958386 I | cephosd: skipping osd.57: "ef7193c1-5e74-4d2f-9a71-6053a96f2c87" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958397 I | cephosd: skipping osd.93: "437d1438-00ca-46d8-bf06-b1969d5c4d25" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958407 I | cephosd: skipping osd.18: "84322b4a-44b2-46cf-b8cb-2d853edaaf07" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958416 I | cephosd: skipping osd.3: "dc0232c3-e516-4062-b67a-c5ec247d4d77" belonging to a different ceph cluster "ae2be176-cf68-4cf0-9f3d-518fe7eac019"
2025-09-19 13:07:26.958424 I | cephosd: 1 ceph-volume raw osd devices configured on this node

I've commented this issue, but maybe it's talos ? or maybe rook issue : rook/rook#15937
I'm on bare metal, obviously I can't delete the disks...

Thanks for your help :)

[smira]

Talos doesn't mark disks for Ceph in any way.

You can use talosctl wipe disk to wipe the signatures.

Also moving /var/lib/rook to be a user volume on a dedicated partition would wipe it as well.

[ktoulliou]

I think is a rook issue. I've found a workaround :

# Debug pod
$ kubectl -n kube-system debug node/myworker1 --image ubuntu --profile sysadmin -it

# Prerequisites : install ceph-volume + ceph-bluestore-tools

# Wipe disk
$ ceph-volume lvm zap /host/dev/sda
stderr: Unknown device "/host/dev/sda": No such device
--> Zapping: /host/dev/sda
--> --destroy was not specified, but zapping a whole device will remove the partition table
--> Removing all BlueStore signature on /host/dev/sda if any...
Running command: /usr/bin/ceph-bluestore-tool zap-device --dev /host/dev/sda --yes-i-really-really-mean-it
Running command: /usr/bin/dd if=/dev/zero of=/host/dev/sda bs=1M count=10 conv=fsync
stderr: 10+0 records in
10+0 records out
stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0680949 s, 154 MB/s
--> Zapping successful for: <Raw Device: /host/dev/sda>

# Restart operator
$ k rollout restart deploy/rook-ceph-operator

Perhaps this is specific to SAS disks? Or is it an incorrect deletion of the Ceph cluster before recreating it?