[BUG] Use of `weakref.proxy` for `rtsys._memsys_module` appears to be incorrect

[cpcloud]

Describe the bug

I am setting up pytest-randomly in order to catch test-order-dependent bugs and this was the first one I found.

When a context is reset, and there's only a single reference to the rtsys._memsys_module object, it gets collected, even though its APIs are still available for use.

The following code is the smallest reproducer I could come up with.

Steps/Code to reproduce bug

import pytest

from numba.cuda.cudadrv import devices
from numba.cuda.memory_management.nrt import rtsys


@pytest.fixture
def alloc_init():
    rtsys.ensure_allocated()
    rtsys.ensure_initialized()


@pytest.fixture
def ctx(alloc_init):
    ctx = devices.get_context()
    yield ctx
    ctx.reset()

    # this fails with a ReferenceError
    str(rtsys._memsys_module)


def test_nothing(ctx):
    """
    The sequence that causes the failure is:

    1. the alloc_init fixture creates a weakref proxy to a CudaPythonModule
       (rtsys._memsys_module) from devices.get_context()
    2. ctx.reset() calls dict.clear() on the context's module dict
    3. since there's only a single strong ref to `rtsys._memsys_module` it gets collected
       during the clear
    4. All ops on the module (a weakref proxy) fail, because it is now
       referencing a dead object
    """

Expected behavior

Since rtsys is a module global, with APIs that liberally use _memsys_module, I would not expect that clearing my current context would immediately make them unusable.

[gmarkall]

Is this the root cause of #517?

[cpcloud]

I think so, yes. That is the same failure mode.

Zooming out, I'm not sure that a weakref here is correct. That said, "just" returning a (non-weakref) object causes other errors that I haven't yet debugged.

From my debugging it looks like the lifetimes are:

0. Program (Python process)
1. Runtime (~Program; it's a singleton)
2. Context (<= Program; generally < Program)
3. Module (<= Context; since they are owned by Context via the modules member)

This means that Runtime holding onto modules would be invalid under these constraints, since Runtime methods potentially use these objects that are liable to be cleaned up by Context while Runtime is still in use or available for use.

Perhaps Modules should be owned by Runtime instead of Context?

[cpcloud]

It turned out that the proxy isn't really being used for anything except module unloading, which isn't called anywhere internally, except in a single test whose last call is module.unload() which will happen anyway when Context goes out of scope.

Ideally we are okay with breaking that API (by removing the ability for modules to unload themselves, which seems fraught given that contexts own them) to fix this bug.

[rparolin]

Looking at the code snippet above... I'm not following the connection between the rtsys the local context object. Why would calling reset on the context object affect a global import at all here?

[cpcloud]

rtsys is a CudaPythonModule, a strong reference to which is held by Contexts in the values of its modules attribute.

[kkraus14]

cc @brandon-b-miller because of numba-cuda runtime and possible cudf exposure