Add async task background worker #4591
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## srothh/worker-class-hierarchy #4591 +/- ##
=================================================================
- Coverage 84.98% 84.48% -0.51%
=================================================================
Files 158 158
Lines 15414 15516 +102
Branches 2432 2448 +16
=================================================================
+ Hits 13100 13108 +8
- Misses 1561 1654 +93
- Partials 753 754 +1
|
srothh
force-pushed
the
srothh/worker-class-hierarchy
branch
from
1261319 to
2896602
Compare
srothh
force-pushed
the
srothh/async-task-worker
branch
from
7ada4b3 to
1a129f7
Compare
srothh
force-pushed
the
srothh/worker-class-hierarchy
branch
from
2896602 to
268ea1a
Compare
srothh
force-pushed
the
srothh/async-task-worker
branch
from
1a129f7 to
97c5e3d
Compare
antonpirker
requested changes
Jul 24, 2025
srothh
force-pushed
the
srothh/worker-class-hierarchy
branch
from
1fbf85f to
ef780f3
Compare
Add a new implementation of the worker interface, implementing the worker as an async task. This is to be used by the upcoming async transport. GH-4581
Refactor the flush method in the async worker to use the async_flush coroutine. GH-4581
…unctions Add a check to see wheter callbacks are awaitable coroutines or functions, as coroutines need to be awaited. GH-4581
…coroutines Coroutines do not return None, therefore it is necessary to consider this in the callback parameter of the worker. Previously, only callbacks with return Type None were accepted. GH-4581
Enable concurrent callbacks on async task worker by firing them as a task rather than awaiting them. A done callback handles the necessary queue and exception logic. GH-4581
Changed kill to also use the _TERMINATOR sentinel, so the queue is still drained to this point on kill instead of cancelled immediately. This should also fix potential race conditions with flush_async. GH-4581
Add proper type annotation to worker task list to fix linting problems GH-4581
Refactor worker implementation to simplify callback processing, fix pending calculation and improve queue initialisation. GH-4581
srothh
force-pushed
the
srothh/async-task-worker
branch
from
97c5e3d to
b5eda0e
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The flush method in the transport enqueues a sync callback for the worker, therefore the check needs to be here after all. GH-4581
antonpirker
approved these changes
Jul 31, 2025
The callbacks passed to the worker from the transport are all async now, so this is currently not needed. GH-4581
|
CI lint issue is in an unrelated file, not sure why it is popping up. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new implementation of the transport background worker based on an async task. This worker mostly mirrors the same functionality as the thread-based worker, with the exception that it exposes a non-blocking async flush (which can be awaited from an async context). Furthermore, the worker itself is not thread-safe and should be called using run_coroutine_threadsafe or similar when called from another thread (this is fixed and handled by the transport). I have kept the fork check from the threaded worker, but I am not sure if it is necessary as forking in an async application would also break the event loop.
GH-4581