Unable to create RAID config on DELL PowerEdge server

[jaxklag]

Hello,

I'm working on Metal3 since a few weeks.

First of all, thanks for this great software and thanks to the community for the work done !

Context

I've started building a LAB with emulated baremetal via libvirt and emulated redfish via Sushy.
All is working as expected. Some features like firmware settings and raid config has not been tested into the LAB.

Now i'm tryng with physical baremetal servers like PowerEdge R660.
Goal is to test "advanced" features like bios settings (works well) and hardware raid config.

I'm trying to create a RAID1 config using BOSS controller on Dell powerEdge bareemtal server.

Here are some info about the environment :

• baremetal model : Dell PowerEedge R660
• BIOS version : 2.3.5
• BMO operator version : quay.io/metal3-io/baremetal-operator:v0.10.3
• Ironic version : quay.io/metal3-io/ironic:v30.0.0

# Ironic version reported by Ironic binary into ironic container
$ kubectl -n baremetal-operator-system exec -it ironic-864b64855f-88v4n -c ironic -- ironic --version
30.0.1.dev1

If i use Redfsh API it's working fine. See command at the end of this message.

If i use Metal3, it doesn't.

Here his the error reported by Ironic during the cleaning phase. Before this error, previous raid volume created manually via iDrac WebUI was deleted by Ironic.

$ kubectl -n baremetal-operator-system logs ironic-864b64855f-88v4n -c ironic |grep -i error

2025-10-08 13:39:07.292 1 DEBUG ironic.drivers.modules.agent_client [None req-78b7ad19-82c4-4395-ba86-3edcb626f44c - - - - - -] Agent command log.collect_system_logs for node a5121476-3cc4-4891-b48c-8b4dcc551465 returned result {'system_logs': '<...>'}, error None, HTTP status code 200 _command /usr/lib/python3.12/site-packages/ironic/drivers/modules/agent_client.py:238
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils [None req-78b7ad19-82c4-4395-ba86-3edcb626f44c - - - - - -] Node a5121476-3cc4-4891-b48c-8b4dcc551465 failed step {'interface': 'raid', 'step': 'create_configuration', 'abortable': False, 'priority': 0}: ("Couldn't find storage by '%(identity)s'", {'identity': 'CPU.1'}): ironic.common.exception.IronicException: ("Couldn't find storage by '%(identity)s'", {'identity': 'CPU.1'})
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils Traceback (most recent call last):
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/conductor/cleaning.py", line 198, in do_next_clean_step
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     result = interface.execute_clean_step(task, step)
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/base.py", line 359, in execute_clean_step
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     return self._execute_step(task, step)
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/base.py", line 321, in _execute_step
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     return getattr(self, step['step'])(task)
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/drac/raid.py", line 157, in create_configuration
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     return super(DracRedfishRAID, self).create_configuration(
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/redfish/raid.py", line 809, in create_configuration
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     logical_disks_to_create = self.pre_create_configuration(
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/drac/raid.py", line 190, in pre_create_configuration
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     storage, controller = DracRedfishRAID._get_storage_controller(
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/drac/raid.py", line 252, in _get_storage_controller
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils     raise exception.IronicException(
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils ironic.common.exception.IronicException: ("Couldn't find storage by '%(identity)s'", {'identity': 'CPU.1'})
2025-10-08 13:39:07.293 1 ERROR ironic.conductor.utils
2025-10-08 13:39:07.785 1 ERROR ironic.conductor.task_manager [None req-78b7ad19-82c4-4395-ba86-3edcb626f44c - - - - - -] Node a5121476-3cc4-4891-b48c-8b4dcc551465 moved to provision state "clean failed" from state "cleaning"; target provision state is "manageable": ironic.common.exception.IronicException: ("Couldn't find storage by '%(identity)s'", {'identity': 'CPU.1'})

I do not understand why it iuse CPU.1 as identity ...

Here are the BMH config and ironic node show outputs :

$ baremetal node show test-metal3~lab-poc-bms1-cp04
+------------------------+-----------------------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                                 |
+------------------------+-----------------------------------------------------------------------------------------------------------------------+
| uuid                   | a5121476-3cc4-4891-b48c-8b4dcc551465                                                                                  |
| created_at             | 2025-10-08T12:53:05.799469+00:00                                                                                      |
| updated_at             | 2025-10-08T13:48:25.580522+00:00                                                                                      |
| automated_clean        | True                                                                                                                  |
| bios_interface         | idrac-redfish                                                                                                         |
| boot_interface         | idrac-redfish-virtual-media                                                                                           |
| boot_mode              | uefi                                                                                                                  |
| clean_step             | {}                                                                                                                    |
| conductor_group        |                                                                                                                       |
| console_enabled        | False                                                                                                                 |
| console_interface      | no-console                                                                                                            |
| disable_power_off      | False                                                                                                                 |
| deploy_interface       | direct                                                                                                                |
| deploy_step            | {}                                                                                                                    |
| description            | None                                                                                                                  |
| driver                 | idrac                                                                                                                 |
| driver_info            | {'redfish_address': 'https://10.9.0.4:443', 'redfish_password': '******', 'redfish_system_id':                        |
|                        | '/redfish/v1/Systems/System.Embedded.1', 'redfish_username': 'bmaas', 'redfish_verify_ca': False, 'deploy_kernel':    |
|                        | 'http://10.9.132.65:6180/images/ironic-python-agent.kernel', 'deploy_ramdisk':                                        |
|                        | 'http://10.9.132.65:6180/images/ironic-python-agent.initramfs'}                                                       |
| driver_internal_info   | {'clean_steps': [{'interface': 'raid', 'step': 'delete_configuration'}, {'interface': 'raid', 'step':                 |
|                        | 'create_configuration'}, {'args': {'settings': [{'name': 'SriovGlobalEnable', 'value': 'Enabled'}, {'name':           |
|                        | 'TimeZone', 'value': 'UTC'}, {'name': 'ErrPrompt', 'value': 'Disabled'}, {'name': 'SysProfile', 'value':              |
|                        | 'PerfOptimized'}, {'name': 'WorkloadProfile', 'value': 'VtOptimizedProfile'}]}, 'interface': 'bios', 'step':          |
|                        | 'apply_configuration'}], 'agent_erase_devices_iterations': 1, 'agent_erase_devices_zeroize': True,                    |
|                        | 'agent_continue_if_secure_erase_failed': False, 'agent_continue_if_ata_erase_failed': False,                          |
|                        | 'agent_enable_nvme_secure_erase': True, 'agent_enable_ata_secure_erase': True, 'disk_erasure_concurrency': 4,         |
|                        | 'agent_erase_skip_read_only': False, 'last_power_state_change': '2025-10-08T13:47:54.995961', 'agent_version':        |
|                        | '11.3.0.dev8', 'agent_last_heartbeat': '2025-10-08T13:38:42.005172', 'hardware_manager_version':                      |
|                        | {'generic_hardware_manager': '1.2'}, 'agent_cached_clean_steps_refreshed': '2025-10-08T13:38:42.051283',              |
|                        | 'cleaning_disable_ramdisk': False, 'agent_secret_token': '******', 'agent_secret_token_pregenerated': '******'}       |
| extra                  | {}                                                                                                                    |
| fault                  | None                                                                                                                  |
| firmware_interface     | redfish                                                                                                               |
| inspection_finished_at | None                                                                                                                  |
| inspection_started_at  | None                                                                                                                  |
| inspect_interface      | agent                                                                                                                 |
| instance_info          | {}                                                                                                                    |
| instance_uuid          | None                                                                                                                  |
| last_error             | None                                                                                                                  |
| lessee                 | None                                                                                                                  |
| maintenance            | False                                                                                                                 |
| maintenance_reason     | None                                                                                                                  |
| management_interface   | idrac-redfish                                                                                                         |
| name                   | test-metal3~lab-poc-bms1-cp04                                                                                         |
| network_data           | {}                                                                                                                    |
| network_interface      | noop                                                                                                                  |
| owner                  | None                                                                                                                  |
| parent_node            | None                                                                                                                  |
| power_interface        | idrac-redfish                                                                                                         |
| power_state            | power on                                                                                                              |
| properties             | {'capabilities': 'boot_mode:uefi,secure_boot:true', 'vendor': 'Dell Inc.'}                                            |
| protected              | False                                                                                                                 |
| protected_reason       | None                                                                                                                  |
| provision_state        | clean wait                                                                                                            |
| provision_updated_at   | 2025-10-08T13:48:25.572089+00:00                                                                                      |
| raid_config            | {'logical_disks': [], 'last_updated': '2025-10-08 13:38:53.166792'}                                                   |
| raid_interface         | idrac-redfish                                                                                                         |
| rescue_interface       | no-rescue                                                                                                             |
| reservation            | None                                                                                                                  |
| resource_class         | None                                                                                                                  |
| retired                | False                                                                                                                 |
| retired_reason         | None                                                                                                                  |
| secure_boot            | False                                                                                                                 |
| service_step           | {}                                                                                                                    |
| shard                  | None                                                                                                                  |
| storage_interface      | noop                                                                                                                  |
| target_power_state     | None                                                                                                                  |
| target_provision_state | manageable                                                                                                            |
| target_raid_config     | {'logical_disks': [{'number_of_physical_disks': 2, 'raid_level': '1', 'size_gb': 447, 'volume_name': 'os'}]}          |
| traits                 | []                                                                                                                    |
| vendor_interface       | idrac-redfish                                                                                                         |
| conductor              | 10.9.132.65                                                                                                           |
| allocation_uuid        | None                                                                                                                  |
| chassis_uuid           | None                                                                                                                  |
+------------------------+-----------------------------------------------------------------------------------------------------------------------+



$ kubectl -n test-metal3 get bmh lab-poc-bms1-cp04
NAME                STATE       CONSUMER   ONLINE   ERROR   AGE
lab-poc-bms1-cp04   preparing              true             59m


$ kubectl -n test-metal3 get bmh lab-poc-bms1-cp04 -oyaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  annotations:
    inspect.metal3.io: disabled
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"metal3.io/v1alpha1","kind":"BareMetalHost","metadata":{"annotations":{"inspect.metal3.io":"disabled"},"name":"lab-poc-bms1-cp04","namespace":"test-metal3"},"spec":{"automatedCleaningMode":"metadata","bmc":{"address":"idrac-virtualmedia+https://10.9.0.4:443/redfish/v1/Systems/System.Embedded.1","credentialsName":"lab-poc-bms1-cp04","disableCertificateVerification":true},"bootMACAddress":"6C:92:CF:7B:D1:00","bootMode":"UEFISecureBoot","online":true,"raid":{"hardwareRAIDVolumes":[{"level":"1","name":"os","numberOfPhysicalDisks":2,"sizeGibibytes":447}]}}}
  creationTimestamp: "2025-10-08T12:53:05Z"
  finalizers:
  - baremetalhost.metal3.io
  generation: 2
  name: lab-poc-bms1-cp04
  namespace: test-metal3
  resourceVersion: "13150357"
  uid: 686094d1-033b-40e4-bf9a-6e331df456c6
spec:
  automatedCleaningMode: metadata
  bmc:
    address: idrac-virtualmedia+https://10.9.0.4:443/redfish/v1/Systems/System.Embedded.1
    credentialsName: lab-poc-bms1-cp04
    disableCertificateVerification: true
  bootMACAddress: 6C:92:CF:7B:D1:00
  bootMode: UEFISecureBoot
  online: true
  raid:
    hardwareRAIDVolumes:
    - level: "1"
      name: os
      numberOfPhysicalDisks: 2
      sizeGibibytes: 447
    softwareRAIDVolumes: null
status:
  errorCount: 3
  errorMessage: ""
  goodCredentials:
    credentials:
      name: lab-poc-bms1-cp04
      namespace: test-metal3
    credentialsVersion: "13142374"
  hardwareProfile: unknown
  lastUpdated: "2025-10-08T13:46:54Z"
  operationHistory:
    deprovision:
      end: null
      start: null
    inspect:
      end: null
      start: null
    provision:
      end: null
      start: null
    register:
      end: "2025-10-08T12:54:36Z"
      start: "2025-10-08T12:53:05Z"
  operationalStatus: OK
  poweredOn: false
  provisioning:
    ID: a5121476-3cc4-4891-b48c-8b4dcc551465
    bootMode: UEFISecureBoot
    image:
      url: ""
    raid:
      hardwareRAIDVolumes:
      - level: "1"
        name: os
        numberOfPhysicalDisks: 2
        sizeGibibytes: 447
      softwareRAIDVolumes: null
    rootDeviceHints:
      deviceName: /dev/sda
    state: preparing
  triedCredentials:
    credentials:
      name: lab-poc-bms1-cp04
      namespace: test-metal3
    credentialsVersion: "13142374"

If i use Redfsh API it's working fine.

# create volume using redfish API
$ curl -skL -u bmaas:<pass> -X POST https://10.9.0.4:443/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes   -H "Content-Type: application/json"   -d '{
    "Name": "os",
    "RAIDType": "RAID1",
    "Drives": [
      { "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.0-0:BOSS.SL.12-1" },
      { "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.1-1:BOSS.SL.12-1" }
    ],
    "VolumeType": "NonRedundant"
  }'

# restart node to apply volume creation task

# Checkvolume is created after reboot
$ curl -skL -u bmaas:<pass> -X GET https://10.9.0.4:443/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1 | jq .
{
  "@Redfish.Settings": {
    "@odata.context": "/redfish/v1/$metadata#Settings.Settings",
    "@odata.type": "#Settings.v1_4_0.Settings",
    "SettingsObject": {
      "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1/Settings"
    },
    "SupportedApplyTimes": [
      "Immediate",
      "OnReset",
      "AtMaintenanceWindowStart",
      "InMaintenanceWindowOnReset"
    ]
  },
  "@odata.context": "/redfish/v1/$metadata#Volume.Volume",
  "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1",
  "@odata.type": "#Volume.v1_10_0.Volume",
  "Actions": {
    "#Volume.CheckConsistency": {
      "@[email protected]": [
        "Immediate",
        "OnReset"
      ],
      "@Redfish.OperationApplyTimeSupport": {
        "@odata.type": "#Settings.v1_4_0.OperationApplyTimeSupport",
        "SupportedValues": [
          "Immediate",
          "OnReset"
        ]
      },
      "[email protected]": [
        "Fast",
        "Slow"
      ],
      "target": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1/Actions/Volume.CheckConsistency"
    },
    "#Volume.Initialize": {
      "@[email protected]": [
        "Immediate",
        "OnReset"
      ],
      "@Redfish.OperationApplyTimeSupport": {
        "@odata.type": "#Settings.v1_4_0.OperationApplyTimeSupport",
        "SupportedValues": [
          "Immediate",
          "OnReset"
        ]
      },
      "[email protected]": [
        "Fast",
        "Slow"
      ],
      "target": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1/Actions/Volume.Initialize"
    }
  },
  "BlockSizeBytes": 512,
  "CapacityBytes": 480036519936,
  "Description": "Virtual Disk 0 on BOSS in SL 12",
  "DisplayName": "os",
  "Encrypted": false,
  "EncryptionTypes": [
    "NativeDriveEncryption"
  ],
  "[email protected]": 1,
  "Id": "Disk.Virtual.0:BOSS.SL.12-1",
  "Identifiers": [],
  "[email protected]": 0,
  "Links": {
    "Drives": [
      {
        "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.0-0:BOSS.SL.12-1"
      },
      {
        "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.1-1:BOSS.SL.12-1"
      }
    ],
    "[email protected]": 2
  },
  "MediaSpanCount": 2,
  "Name": "os",
  "Oem": {
    "Dell": {
      "@odata.type": "#DellVolume.v1_0_0.DellVolume",
      "DellVirtualDisk": {
        "@odata.context": "/redfish/v1/$metadata#DellVirtualDisk.DellVirtualDisk",
        "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Volumes/Disk.Virtual.0:BOSS.SL.12-1/Oem/Dell/DellVolumes/Disk.Virtual.0:BOSS.SL.12-1",
        "@odata.type": "#DellVirtualDisk.v1_2_0.DellVirtualDisk",
        "BusProtocol": "PCIE",
        "Cachecade": "NonCachecadeVD",
        "Description": "An instance of DellVirtualDisk will have data specific to the Virtual Disks in the system.",
        "DiskCachePolicy": "Default",
        "Id": "Disk.Virtual.0:BOSS.SL.12-1",
        "LastSystemInventoryTime": "2025-10-08T14:15:23+00:00",
        "LastUpdateTime": "2025-10-08T14:15:21+00:00",
        "LockStatus": "Unlocked",
        "MediaType": "SolidStateDrive",
        "Name": "DellVirtualDisk",
        "ObjectStatus": "Current",
        "OperationName": "None",
        "OperationPercentComplete": 0,
        "PrimaryStatus": "OK",
        "RaidStatus": "Online",
        "ReadCachePolicy": "NoReadAhead",
        "RemainingRedundancy": 1,
        "SpanDepth": 1,
        "SpanLength": 2,
        "StartingLBAinBlocks": 0,
        "StripeSize": "128KB",
        "T10PIStatus": "Disabled",
        "VirtualDiskTargetID": 0,
        "WriteCachePolicy": "WriteThrough"
      },
      "[email protected]": "Please migrate to use DellVolume",
      "DellVolume": {
        "BusProtocol": "PCIE",
        "Cachecade": "NonCachecadeVD",
        "DiskCachePolicy": "Default",
        "[email protected]": [
          "Default",
          "Disabled",
          "Enabled"
        ],
        "LastSystemInventoryTime": "2025-10-08T14:15:23+00:00",
        "LastUpdateTime": "2025-10-08T14:15:21+00:00",
        "LockStatus": "Unlocked",
        "MediaType": "SolidStateDrive",
        "ObjectStatus": "Current",
        "OperationName": "None",
        "OperationPercentComplete": 0,
        "PrimaryStatus": "OK",
        "RaidStatus": "Online",
        "RemainingRedundancy": 1,
        "SpanDepth": 1,
        "SpanLength": 2,
        "StartingLBAinBlocks": 0,
        "StripeSize": "128KB",
        "T10PIStatus": "Disabled",
        "VirtualDiskTargetID": 0
      }
    }
  },
  "Operations": [],
  "[email protected]": 0,
  "OptimumIOSizeBytes": 131072,
  "RAIDType": "RAID1",
  "ReadCachePolicy": "Off",
  "[email protected]": [
    "Off"
  ],
  "Status": {
    "Health": "OK",
    "HealthRollup": "OK",
    "State": "Enabled"
  },
  "VolumeType": "Mirrored",
  "WriteCachePolicy": "WriteThrough",
  "[email protected]": [
    "WriteThrough"
  ]
}

I appreciate the help the community can provide me to understand why it's not working.

If you need more traces or log tell me.

Thanks

[tuminoid]

Hello @jaxklag ! I see you closed the discussion before anyone responded. If you solved the issue, could you detail here what the issue was for the next person ?

[jaxklag]

Hello @tuminoid
I think "discussions" was not the good place.
I've just opened an issue : #2720

Regards,