Loop during device probing causing high CPU

[adamoswick]

What steps did you take and what happened:
When an unsupported device type is probed for data (e.g. smart), NDM seems to loop continuously, retrying without any delay.

This leads to NDM using an entire CPU core all the time. Once the 'unsupported device' (in this case /dev/vdb) is removed from the include filter and the pod is restarted, CPU usage returns to normal.

What did you expect to happen:
NDM to either stop probing for information or pause before retrying to avoid high CPU usage.

The output of the following commands will help us better understand what's going on:

Logs from NDM

# kubectl logs --tail=40 -n openebs openebs-ndm-dzspt
I0621 16:58:36.783757       6 udevprobe.go:245] Dependents of /dev/loop5 : {Parent: Partitions:[] Holders:[] Slaves:[]}
I0621 16:58:36.783786       6 udevprobe.go:255] Device: /dev/loop5 is of type: loop
I0621 16:58:36.784096       6 udevprobe.go:245] Dependents of /dev/loop6 : {Parent: Partitions:[] Holders:[] Slaves:[]}
I0621 16:58:36.784125       6 udevprobe.go:255] Device: /dev/loop6 is of type: loop
I0621 16:58:36.784472       6 udevprobe.go:245] Dependents of /dev/loop7 : {Parent: Partitions:[] Holders:[] Slaves:[]}
I0621 16:58:36.784490       6 udevprobe.go:255] Device: /dev/loop7 is of type: loop
I0621 16:58:37.272347       6 blockdevicestore.go:131] eventcode=ndm.blockdevice.deactivate.success msg=Deactivated blockdevice rname=blockdevice-f71acf94b0f36750c26e473673f326c9
I0621 16:58:37.277559       6 eventhandler.go:69] Processing details for /dev/vda
I0621 16:58:37.277883       6 probe.go:119] details filled by udev probe
I0621 16:58:37.278067       6 sysfsprobe.go:97] blockdevice path: /dev/vda capacity :10737418240 filled by sysfs probe.
I0621 16:58:37.278182       6 sysfsprobe.go:125] blockdevice path: /dev/vda logical block size :512 filled by sysfs probe.
I0621 16:58:37.278280       6 sysfsprobe.go:137] blockdevice path: /dev/vda physical block size :512 filled by sysfs probe.
I0621 16:58:37.278392       6 sysfsprobe.go:149] blockdevice path: /dev/vda hardware sector size :512 filled by sysfs probe.
I0621 16:58:37.278499       6 sysfsprobe.go:160] blockdevice path: /dev/vda drive type :HDD filled by sysfs probe.
I0621 16:58:37.278510       6 probe.go:119] details filled by sysfs probe
E0621 16:58:37.278621       6 smartprobe.go:101] map[errorCheckingConditions:the device type is not supported yet, device type: "unknown"]
I0621 16:58:37.278652       6 probe.go:119] details filled by smart probe
I0621 16:58:37.279155       6 mountprobe.go:134] no mount point found for /dev/vda. clearing mount points if any
I0621 16:58:37.279167       6 probe.go:119] details filled by mount probe
I0621 16:58:37.281577       6 probe.go:119] details filled by blkid probe
I0621 16:58:37.281594       6 usedbyprobe.go:122] device: /dev/vda is not having any zfs partitions
I0621 16:58:37.281703       6 probe.go:119] details filled by used-by probe
I0621 16:58:37.281719       6 probe.go:119] details filled by Custom Tag Probe
I0621 16:58:37.281724       6 addhandler.go:53] device: /dev/vda does not exist in cache, the device is now connected to this node
I0621 16:58:37.281818       6 osdiskexcludefilter.go:131] applying os-filter regex ^/dev/loop0(p[0-9]+)?$ on /dev/vda
I0621 16:58:37.281907       6 osdiskexcludefilter.go:131] applying os-filter regex ^/dev/vda[0-9]*$ on /dev/vda
I0621 16:58:37.281932       6 filter.go:89] /dev/vda ignored by os disk exclude filter
I0621 16:58:37.281986       6 eventhandler.go:69] Processing details for /dev/vda1
I0621 16:58:37.282275       6 probe.go:119] details filled by udev probe
I0621 16:58:37.282441       6 sysfsprobe.go:97] blockdevice path: /dev/vda1 capacity :104857600 filled by sysfs probe.
I0621 16:58:37.282564       6 sysfsprobe.go:125] blockdevice path: /dev/vda1 logical block size :512 filled by sysfs probe.
I0621 16:58:37.282654       6 sysfsprobe.go:137] blockdevice path: /dev/vda1 physical block size :512 filled by sysfs probe.
I0621 16:58:37.282753       6 sysfsprobe.go:149] blockdevice path: /dev/vda1 hardware sector size :512 filled by sysfs probe.
I0621 16:58:37.282833       6 sysfsprobe.go:160] blockdevice path: /dev/vda1 drive type :HDD filled by sysfs probe.
I0621 16:58:37.282839       6 probe.go:119] details filled by sysfs probe
E0621 16:58:37.282868       6 smartprobe.go:101] map[errorCheckingConditions:the device type is not supported yet, device type: "unknown"]
I0621 16:58:37.282878       6 probe.go:119] details filled by smart probe
I0621 16:58:37.283289       6 mountprobe.go:134] no mount point found for /dev/vda1. clearing mount points if any
I0621 16:58:37.283306       6 probe.go:119] details filled by mount probe
I0621 16:58:37.283311       6 probe.go:119] details filled by blkid probe

OpenEBS pods

# kubectl get pods -n openebs
NAME                                           READY   STATUS    RESTARTS      AGE
openebs-localpv-provisioner-74d646c967-h7z27   1/1     Running   2 (15m ago)   2d19h
openebs-ndm-dzspt                              1/1     Running   0             12m
openebs-ndm-hgjz6                              1/1     Running   1 (15m ago)   2d3h
openebs-ndm-hxj75                              1/1     Running   1 (15m ago)   2d3h
openebs-ndm-operator-69b5fcf66b-9vtwp          1/1     Running   2 (15m ago)   2d19h

Block device in question

apiVersion: openebs.io/v1alpha1
kind: BlockDevice
metadata:
  annotations:
    internal.openebs.io/uuid-scheme: gpt
  creationTimestamp: "2022-06-18T21:20:36Z"
  generation: 6
  labels:
    kubernetes.io/hostname: talos-192-168-40-38
    ndm.io/blockdevice-type: blockdevice
    ndm.io/managed: "true"
    nodename: talos-192-168-40-38
  name: blockdevice-f71acf94b0f36750c26e473673f326c9
  namespace: openebs
  resourceVersion: "209073"
  uid: ce2a901f-1397-4d9b-a6d3-4acf8ec1baeb
spec:
  capacity:
    logicalSectorSize: 512
    physicalSectorSize: 512
    storage: 1030792116736
  details:
    compliance: ""
    deviceType: partition
    driveType: HDD
    firmwareRevision: ""
    hardwareSectorSize: 512
    logicalBlockSize: 512
    model: ""
    physicalBlockSize: 512
    serial: ""
    vendor: ""
  devlinks: []
  filesystem:
    fsType: xfs
    mountPoint: /var/mnt/openebs
  nodeAttributes:
    nodeName: talos-192-168-40-38
  partitioned: "No"
  path: /dev/vdb1
status:
  claimState: Unclaimed
  state: Inactive

Environment:

• OpenEBS version: 3.2.0
• Kubernetes version: v1.23.6
• OS: Talos v1.0.6
• Type of disks connected to the nodes: Virtual disks

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/talos-dev-005-data.qcow2' index='2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

[vchrisb]

I do have the same issue. For me it was the newish /dev/zram devices.