check_fs_used reporting issues on wrong filesystem

[eliv]

Using nhc 1.4.2, I have several check_fs_used for our grid nodes. Works great most of the time, auto draining the nodes when the utilization gets to high. However, when a specific filesystem is getting hammered and slow to respond the check_fs_used() tends to report an issue with the first filesystem called with check_fs_used(), not the specific one that's slow to respond. This appears to be because it's not calling df on only the filesystem that's being tested but all filesystems every time check_fs_used() is called.

[eric4tech]

I second this issue. The fact that any check_fs_* does a blanket df causes a problem if there is ANY mount slow to respond. This results in the check being reported as failed and drains the nodes (if enabled).

I'm looking now to determine the best way to just pass the filesystem in question to the df command within lbnl_fs.nhc so that it only runs df against the filesystems that I want to check. To be honest, I don't feel the need to check remote filesystems for inodes/free space as my storage servers should be the ones informing me of issues there. I'm mostly concerned with the local disks.

I already modified the df command to run 'df -Tk' instead of 'df -Tka' as per issue #44 but I suppose I can just add a '-t ext4' to only run against my locals. This obviously isn't globally acceptable, though.

As a global solution, I would suggest passing the file system to be checked into the df command within the lbnl_fs.nhc script. i.e. where the config is set as 'check_fs_free /local 5%', this should effectively run 'df -Tk /local' to get the desired results then parse only that data.

[eric4tech]

actually, 'df -Tk -t ext4' does not work as I expected when a filesystem is slow to respond. It times out as well. I guess the filesystem to be checked needs to be passed directly to df, as that does work. ie 'df -Tk /local'. I haven't figured out yet what to change in the code to get this to work...

[mej]

The behavior you're describing is by design. NHC tries to minimize to the greatest extent possible the number of external commands it executes in order to, in turn, minimize the chances of getting hung up due to shared filesystems going away, for example. It's optimized mainly for the statelessly provisioned cluster node case, where the root filesystem is frequently mounted in part, or in full, via NFS.

I'll certainly grant you that this isn't always going to be the case, and perhaps it would make sense to make the behavior optional. But this will likely need to wait for 1.4.4; I'm trying to get the remaining known bugs/issues resolved so I can get the way-too-long-delayed 1.4.3 release out the door before the sun goes nova....