efa: Expose 128 bytes data polling device capability

[dkkranz]

EFA can support 128 bytes blocks write in-order for some Grace platforms. Move the check on x86 architecture to the mlx5 provider implementation.

Reviewed-by: Michael Margolin [email protected]
Reviewed-by: Yonatan Nachum [email protected]

[jgunthorpe]

I don't think this is right, it would break things like PPC. The platform check is supposed to be in the common code, and the driver check is only about how the device works on PCI.

You should probably just add ARM to the inclusion list, though I imagine there are some wonky ARMs that don't work, they may not matter in practice.

[dkkranz]

Applied your suggestion, thanks

[dkkranz]

@jgunthorpe Kind reminder

[jgunthorpe]

I don't know.. I just feel uneasy about this. ARM is such a wide and varied architecture I don't know if we can really say it is actually reliably in order. Grace might be just because it has 128 byte lines, but that doesn't mean a different ARM chip will be. This could break some of the MPIs running on ARM supercomputers.. Do you think you could limit it just to Grace? Did someone from NVIDIA confirm that this is actually true on Grace?

[jgunthorpe]

I consulted with people here and unfortunately I don't think using ARM platform as a condition is sufficient. Even looking at NVIDIA CPUs this only works on certain CPUs, with certain configurations.

I guess EFA could be special since it really only exists in AWS instances and you guys will do the validation. I'd still strongly recommend that you query your device FW to see if it is supported as I'm not sure you will find all future systems will work this way.

So maybe the right thing is to add a new op query_qp_data_in_order_force() or something like that that skips the architecture check in the core code. I don't really want to have drivers doing architecture checks..

[dkkranz]

@jgunthorpe Please review the last revision that adds a flag to skip the architecture check.

[jgunthorpe]

I didn't mean a flag to userspace, there is nothing userspace can do with this. I ment some kind of flag from EFA to the core code to disable the arch checks in the core code just for EFA.

[mrgolin]

I don't really want to have drivers doing architecture checks..

But this is exactly what you are suggesting now, and I'm not sure why we need to have architecture checks in libibverbs either.

I think there is use for a common way to query whether data is written in order from device perspective only, for instance data polling from GPU isn't necessarily affected by the CPU architecture. Don't you agree?

[dkkranz]

@jgunthorpe Was your concern addressed by @mrgolin on his last comment? Do we have any blocker for merging this?

[jgunthorpe]

No, I don't want architecture checks in drivers. EFA is being very weird and special here by saying that since EFA is only part of a fully engineered cloud system it can know that it only exists in conjunction with, otherwise architecture undefined, behavior.

I think what you want is to just make EFA do this unconditionally so EFA and only EFA would skip the architecture checks in the core code. If you ever deploy EFA on some implementation that doesn't work that way then EFA should report this to the driver through the FW.

[jgunthorpe]

Maybe you could reorganize things so the driver calls out some kind of core helper to test if the CPU/platform is OK for some properties and then just not call that on EFA.

[dkkranz]

The EFA capability will be exposed via direct EFA device attribute, without changing the general API.
Please review the changes.