mstflint v4.33 unable to query a device in kernel lockdown

[markhaywood]

When trying to query an mlx5_0 device while the system is in kernel lockdown, fails with v4.33:

$ ibdev2netdev -v
0000:af:00.0 bnxt_re0 (MT5847 - Broadcom ) Oracle Dual Port 25Gb Ethernet Adapter fw 214.2.255.0 port 1 (DOWN ) ==> ens1f0np0 (Down)
0000:af:00.1 bnxt_re1 (MT5847 - Broadcom ) Oracle Dual Port 25Gb Ethernet Adapter fw 214.2.255.0 port 1 (DOWN ) ==> ens1f1np1 (Down)
0000:3b:00.0 mlx5_0 (MT4121 - 7359059 ) CX556A - ConnectX-5 QSFP28 -Aux[1] fw 16.35.4506 port 1 (ACTIVE) ==> ens2f0np0 (Up)
0000:3b:00.1 mlx5_1 (MT4121 - 7359059 ) CX556A - ConnectX-5 QSFP28 -Aux[1] fw 16.35.4506 port 1 (ACTIVE) ==> ens2f1np1 (Up)

$ mstconfig -d mlx5_0 query
-E- Failed to open the device

$ mstconfig -d 3b:00.0 query
-E- Failed to identify the device

This works fine with mstflint 4.32:

$ mstconfig -d mlx5_0 query

Device #1:

Device type: ConnectX5
Name: 7359059_OCI_SD_Ax_Bx
Description: ConnectX-5 Ex VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe4.0 x16; tall bracket; ROHS R6
Device: mlx5_0

Configurations: Next Boot
FLEX_PARSER_PROFILE_ENABLE 0

  ...

    SAFE_MODE_ENABLE                            True(1) 

$ mstconfig -d 3b:00.0 query

Device #1:

Device type: ConnectX5
Name: 7359059_OCI_SD_Ax_Bx
Description: ConnectX-5 Ex VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe4.0 x16; tall bracket; ROHS R6
Device: mlx5_0

Configurations: Next Boot
FLEX_PARSER_PROFILE_ENABLE 0

    ...

    SAFE_MODE_ENABLE                            True(1) 

Debugging this a bit, it appears to me that this was broken by patches to mtcr_parse_name() in mtcr_ul/mtcr_ul_com.c such as:

3b1d749

[markhaywood]

This fixes the problem. But I'm not sure if it breaks the intent of 3b1d749

diff --git a/mtcr_ul/mtcr_ul_com.c b/mtcr_ul/mtcr_ul_com.c
index e1bae586f58f..683c5b5b3d8e 100644
--- a/mtcr_ul/mtcr_ul_com.c
+++ b/mtcr_ul/mtcr_ul_com.c
@@ -2606,7 +2606,7 @@ static MType mtcr_parse_name(const char* name,
         return MST_FWCTL_CONTROL_DRIVER;
     }
 
-    if (is_vfio || lockdown_enabled) {
+    if (is_vfio && lockdown_enabled) {
         if (is_vfio) {
             scnt = sscanf(name, "vfio-%x:%x:%x.%x", &my_domain, &my_bus, &my_dev, &my_func);
             if (scnt != 4) {

[itayav17]

Hi @markhaywood ,
In a kernel lockdown environment the only way to access the device is through vfio (or through the mstflint driver if it runs in kernel space).
After the last patch to detect kernel lockdown, does it work for you?

[markhaywood]

We have always used the mstflint_access module to access the device in kernel lockdown. It no longer works in v4.33. This appears to be a result to changes made to mtcr_parse_name(). Clearly, in v4.33 specifying the device by name "mlx5_0" has been broken. But even if we use the PCI ID, we see errors as demonstrated above.

If there are patches to v4.33 that you want me to try, please list them.

[itayav17]

@markhaywood thanks for the clarification. We will check internally and keep you posted.

[itayav17]

Hi @markhaywood , when you use ibdev2netdev, you probably need the NIC driver to be loaded. in case the mlx5 is attached to the device, the vfio cannot work.

[markhaywood]

@itayav17 I'm not sure what ibdev2netdev has to do with this?

On a system with lockdown enabled and mstflint_access loaded, if I install v4.33 (built from the v4.33.0-1 tag) and run the command:

$ mstconfig -d mlx5_0 query
-E- Failed to open the device

If I then uninstall v4.33 and install v4.32 (built from the v4.32.0-1 tag) and re-run the command, I see the expected output. I then uninstall v4.32 and install v4.33 and re-run the command and again the failure.

A gdb backtrace of the v4.33 failure looks like this:

#0  0x000055555557e940 in mtcr_parse_name ()
#1  0x0000555555580729 in mopen_ul_int ()
#2  0x0000555555578706 in mopen_adv ()
#3  0x000055555543dcdb in Commander::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Device_Type) ()
#4  0x00005555554352ca in MlxCfg::queryDevCfg(char const*, char const*, int, bool) ()
#5  0x0000555555435575 in MlxCfg::queryDevsCfg() ()
#6  0x0000555555424ce3 in main ()

If I modify mtcr_parse_name():

--- a/mtcr_ul/mtcr_ul_com.c
+++ b/mtcr_ul/mtcr_ul_com.c
@@ -2606,6 +2606,7 @@ static MType mtcr_parse_name(const char* name,
         return MST_FWCTL_CONTROL_DRIVER;
     }
 
+    fprintf(stderr, "mtcr_parse_name name = %s\n", name);
     if (is_vfio || lockdown_enabled) {
         if (is_vfio) {
             scnt = sscanf(name, "vfio-%x:%x:%x.%x", &my_domain, &my_bus, &my_dev, &my_func);
@@ -2622,6 +2623,7 @@ static MType mtcr_parse_name(const char* name,
                 my_domain = 0;
                 scnt = sscanf(name, "%x:%x.%x", &my_bus, &my_dev, &my_func);
                 if (scnt != 3) {
+                   fprintf(stderr, "mtcr_parse_name exiting here with MST_ERROR\n");
                     return MST_ERROR;
                 }
             }

$ mstconfig -d mlx5_0 query
mtcr_parse_name name = mlx5_0
mtcr_parse_name exiting here with MST_ERROR
-E- Failed to open the device

Clearly this is always going to fail parsing the device by name. And even when parsed by PCI id, this no longer works even with mstflint_accesss loaded:

$ mstconfig -d 3b:00.0 query
mtcr_parse_name name = 3b:00.0
Error: Failed to open new_id file: /sys/bus/pci/drivers/vfio-pci/new_id
write offset: Bad file descriptor
FATAL - crspace read (0xf0014) failed: Bad file descriptor
-E- Failed to identify the device

[markhaywood]

This issue no longer exists for me in v4.34 as a result of dde11c9 ("Add --enable-vfio configure flag to make VFIO library optional"). In the case of our build, this conditional removes the offending code in mtcr_parse_name() from the source that is built.