Skip to content

feat: processors async - #1006

Draft
kaya-david wants to merge 12 commits into
mainfrom
feat-processors-async
Draft

feat: processors async#1006
kaya-david wants to merge 12 commits into
mainfrom
feat-processors-async

Conversation

@kaya-david

@kaya-david kaya-david commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Description

... work in progress ...

  • geoip_enricher

    • make database loading and processing async
    • use async file locking and offload blocking filesystem operations
    • adapt tests
  • grokker

    • load custom grok patterns asynchronously during setup
    • offload blocking downloads, ZIP extraction, and pattern initialization
    • clean up temporary pattern directories asynchronously
  • requester

    • use aiohttp for asynchronous HTTP requests
    • manage the HTTP client session during setup and shutdown
    • preserve existing request configuration through aiohttp parameter conversion
    • adapt tests to use an async HTTP test server
  • amides

    • load models asynchronously during setup
    • offload blocking filesystem operations, downloads, ZIP access, and model deserialization
    • prepare model downloads for asynchronous getter support

Assignee

  • The changes adhere to the contribution guidelines
  • I have performed a self-review of my code
  • My changes generate no new warnings (e.g. flake8/mypy/pytest/...) other than deprecations

Documentation

Code Quality

  • Patch test coverage > 95% and does not decrease
  • New code uses correct & specific type hints

How did you verify that the changes work in practice?

  • pytest

Reviewer


The rendered docs for this PR can be found here.

@kaya-david kaya-david changed the title fet: processors async feat: processors async Aug 5, 2026
@kaya-david
kaya-david force-pushed the feat-processors-async branch 7 times, most recently from 757fe41 to 1752027 Compare August 6, 2026 08:37
@codecov-commenter

codecov-commenter commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.04%. Comparing base (ce970b7) to head (7f7a0a2).

Files with missing lines Patch % Lines
logprep/ng/processor/requester/processor.py 72.41% 24 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1006      +/-   ##
==========================================
- Coverage   93.16%   93.04%   -0.12%     
==========================================
  Files         234      234              
  Lines       16070    16169      +99     
==========================================
+ Hits        14972    15045      +73     
- Misses       1098     1124      +26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kaya-david
kaya-david force-pushed the feat-processors-async branch from 1752027 to fe6a7f9 Compare August 12, 2026 06:56
@kaya-david kaya-david self-assigned this Aug 12, 2026
@kaya-david
kaya-david force-pushed the feat-processors-async branch from 4bbc458 to a791b13 Compare August 13, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants