Testing GPU support -- possible solution

[ronvree]

I think I found something interesting regarding the implementation/testing of cpu/gpu support: apparently PyTorch recently introduced a "meta" device that can be used exactly for testing these kind of purposes (https://docs.pytorch.org/docs/stable/meta.html). If I understood correctly, it allows you to move tensors to a "meta" device which is just some hypothetical imaginary device that stores no data. It also allows you create "meta" computations without actual data and can be used to verify whether the device switching happened properly. If the model attempts to make a computation using both a "meta" and cpu/gpu tensor it crashes. So I think if a model can successfully be transferred from cpu to a meta device and ran there it guarantees compatibility with gpu. (if its available of course). And this test can be run without the use of a gpu! Anyways curious to hear your thoughts!

[SarahAlidoost]

So I think if a model can successfully be transferred from cpu to a meta device and ran there it guarantees compatibility with gpu.

hmmm, not really. I did a quick look into the documentation and this open issue on pytorch. It seems that by using device "meta", we cannot catch specific errors and bugs on gpu device. The "meta" device can be useful only for shapes and dtype correctness and for lazy loading. For testing purposes there is another module FakeTensorMode which is able to simulate actual device semantics without being on gpu. Both are something to explore.
cc @fnattino and @SCiarella

[ronvree]

what do you mean with bugs on the gpu device? like issues with numerical precision? I agree no numerical results can be verified but for creating a separate test for device compatibility it should suffice right?

[ronvree]

ah actually memory related issues wouldn't be caught with meta tensors

[ronvree]

although this is something that can maybe be tested separately

[SarahAlidoost]

what do you mean with bugs on the gpu device? like issues with numerical precision?

yes, numerical precision is one of them, others like memory error, cuda kernel issues if any, cannot be tested with "meta". But definitely, something to look into it.