Remediation to the function of .wait()

[rherriott]

Which project are you reporting a bug for?

kr8s

What happened?

Often, when I use .wait() (typically async_wait), I'm effectively waiting on "being ready".
For example, I can create a Pod, wait on that pod being ready via 'condition=Ready'

This currently does not reliably work, and gets worse when I'm waiting on something like '.status.readyToUse'.

As a result, I'm stuck either while-try-looping around each .wait(), or manufacturing it myself via

while 'a' not in object or 'b' not in object.a  or object.a.b != value:
    await asyncio.sleep(3.0)
    await object.async_refresh()

As .wait() throws an error when '.status' doesn't exist, or when .readyToUse doesn't exist in '.status'.

I can see having some sort of option to turn on failing if there's no value present in the jsonpath case, but the default usage of .wait() being 'no '.status'? exception.' is nonsensical.

The quickest solution is to edit the line in _objects.py -> APIObject._test_conditions() that calls jsonpath.findall to catch the absence at least of .status

Also, the docs don't mention jsonpath existing in wait()

Thanks!
It's really a good bit of software, that's why I use it so much after all 👍 👍

Anything else?

No response

[jacobtomlinson]

As .wait() throws an error when '.status' doesn't exist, or when .readyToUse doesn't exist in '.status'.

Can you share a minimal snippet of code that reproduces this error? I'd like to see it for myself to figure out the best solution.

The quickest solution is to edit the line in _objects.py -> APIObject._test_conditions() that calls jsonpath.findall to catch the absence at least of .status

Can you share the edit you are making? Maybe we want to add this permanently?

Also, the docs don't mention jsonpath existing in wait()

Good point. We should fix this.

It's really a good bit of software, that's why I use it so much after all 👍 👍

🚀

[rherriott]

When I run

VSObject = kr8s.objects.new_class(kind='VolumeSnapshot', version='snapshot.storage.k8s.io/v1', namespaced=True)
vs = VSObject(...)
await vs.async_create()
await vs.async_wait("jsonpath='{.status.readyToUse}'='True'")

In a bunch of threads, (> 10) the likelihood that one of them throws is high.

[jacobtomlinson]

I'm a little nervous that you are mixing sync and async APIs together. This isn't a use case we actively support. If you just use the async API do you still see this issue?

VSObject = kr8s.asyncio.objects.new_class(kind='VolumeSnapshot', version='snapshot.storage.k8s.io/v1', namespaced=True)
vs = VSObject(...)
await vs.create()
await vs.wait("jsonpath='{.status.readyToUse}'='True'")

If you do still see it can you provide me with a minimal snippet that I can copy/paste to reproduce this? Pseudo code with ... and sections like running in many theads omitted means I have to try and figure out what you are doing to see it for myself.

[rherriott]

I'll try to provide a type-consistent and more-reproducible example for clarity. However, as far as I can tell, calling <funcname>() differs between sync an async versions of a class (sync wraps, async just calls async_<funcname>()), but calling async_<funcname>() always gets to the same function, no?

Nonetheless what I provided is insufficient, as the fact that it was taking place in an undescribed async environment requires more explicit explication, which follows.

import asyncio, kr8s

SOURCE_NS = 'namespace-a'
TARGET_NS = 'namespace-b'

worklist = [
    ("pvc-00", "pvc-new-00", "20Gi"),
    ("pvc-01", "pvc-new-01", "30Gi"),
    ("pvc-02", "pvc-new-02", "10Gi"),
    ("pvc-03", "pvc-new-03", "150Gi"),
    ("pvc-04", "pvc-new-04", "5Gi"),
    ("pvc-05", "pvc-new-05", "20Gi"),
    ("pvc-06", "pvc-new-06", "22Gi"),
    ("pvc-07", "pvc-new-07", "4Gi"),
    ("pvc-08", "pvc-new-08", "90Gi"),
    ("pvc-09", "pvc-new-09", "2Gi"),
    ("pvc-10", "pvc-new-10", "6Gi"),
    ("pvc-11", "pvc-new-11", "80Gi"),
    ("pvc-12", "pvc-new-12", "45Gi")
]

async def handle(source_pvc_name: str, source_pvc_namespace: str, target_pvc_name: str, target_pvc_namespace: str, size: str) -> None:
    api = kr8s.api()
    VSObject = kr8s.asyncio.objects.new_class(kind='VolumeSnapshot', version='snapshot.staorage.k8s.io/v1', namespaced=True)
    vs_name = ('s-' + source_pvc_name)[-63:]
    vs = VSObject({
        'apiVersion': 'snapshot.storage.k8s.io/v1',
        'kind': 'VolumeSnapshot',
        'metadata': {
            'name': vs_name
        },
        'spec': {
            'volumeSnapshotClassName': 'snapclass',
            'source': {
                'persistentVolumeClaimName': source_pvc_name
            }
        }
    }, namespace=source_pvc_namespace, api=api)
    await vs.create()
    await vs.refresh()
    await vs.wait("jsonpath='{.status.readyToUse}'='True'")
    return

async def act() -> None:
    futures = []
    for line in worklist:
        futures.append(asyncio.create_task(handle(line[0], SOURCE_NS, line[1], TARGET_NS, line[2])))
    await asyncio.gather(*futures)
    pass

if __name__ == "__main__":
  asyncio.run(act())

This is as reduced as I can make the code while still maintaining the core function. Of note: I'm running this on Windows, and the target is on AWS EKS. Perhaos if this is run against something local the updates are quicker, dunno.

[rherriott]

Of note: the occasional error this gives is either that .status does not exist, or that .readyToUse does not exist within .status.

[jacobtomlinson]

but calling async_() always gets to the same function, no?

This it true, and you're more than welcome to call the async_ methods directly if you prefer. My bigger concern is that you're using the sync version of new_class, the classes that it generates have the _async property set to False which can invoke the sync wrapping machinery which causes background threads to be created.

By using sync objects there are situations where you could accidentally call something which causes a coroutine to be dispatched to the event loop running in the thread instead of the main event loop. If you're using it with async you never want this to happen as you may end up with objects being used across two different event loops. By using the async API this can't happen because the sync event loop is never started, you'll just get an error.

These kind of bugs are hard to pin down, and I was wondering if that was what is happening here. Again in the example you gave you're creating a sync API with api = kr8s.api(). Please use api = await kr8s.asyncio.api().

Of note: the occasional error this gives is either that .status does not exist, or that .readyToUse does not exist within .status.

Can you also share these errors with the full traceback?

[rherriott]

That's very interesting! I'll go and see whether it's possible to get the same error with the asyncio api.