[FEA] Implement validation of memory resource passing #20626
Description
Is your feature request related to a problem? Please describe.
We have some strict testing in place to ensure that all libcudf (and soon, pylibcudf) APIs that accept streams actually forward those streams to all corresponding CUDA calls. APIs that accept memory resources are also expected to follow similarly strict rules where all output data should be allocated using that memory resource while all temporary memory during the algorithm should use cudf's current device resource. However, we currently have no testing or enforcement of that.
Describe the solution you'd like
We should implement a testing framework to validate that memory resources are always used as intended. A solution is much easier to implement in pylibcudf than in libcudf since, as for streams, pylibcudf has its own default memory resources independent of libcudf's that we can directly monkey-patch independently of the default mr libcudf will use for temporary allocations. The approach I would suggest is to use rmm's statistics mr around both the memory resources to track allocations. That will allow us to validate that all final allocations are performed on the passed-in memory resource by checking that the final total allocation on cudf's default mr is 0 (i.e. it was only used for temporaries, which have been freed). Ensuring that temporary allocations occurred on the other mr is harder. One possibility would be to try to check whether the maximum allocation on the passed-in mr is every larger than the total output memory, since that would indicate that we allocated something we didn't need to. This approach is less airtight, but would be a much better start than nothing.